Systems and methods for semantic paraphrasing

ABSTRACT

The presently disclosed embodiments may include a computer readable medium including instructions that when executed by one or more processing devices cause the one or more processing devices to perform a method. The method may include: analyzing an electronic document text; identifying in the electronic document text a first drafted text element to be modified; causing the first drafted text element to be highlighted to a user on a display; generating at least one text output option that conveys a meaning associated with the first drafted text element but includes one or more changes relative to the first drafted text element; and causing the at least one text output option to be shown to the user via the display.

RELATED APPLICATIONS

This application is a continuation of PCT International Application No.PCT/2020/041846, filed Jul. 13, 2020, which claims priority from U.S.Provisional Patent Application No. 62/882,732, filed on Aug. 5, 2019;U.S. Provisional Patent Application No. 62/882,734, filed on Aug. 5,2019; and U.S. Provisional Patent Application No. 62/943,493, filed onDec. 4, 2019. The entire disclosure of each are hereby incorporated byreference in the present application.

BACKGROUND

The disclosed technology relates generally to controllable naturallanguage generation from an automated computer-based system. Priorsystems can generate text, for example, based on words a user haspreviously typed. These prior systems, however, often rely onprobabilities associated with the user's typing habits, or they may relyon statistical models that analyze the probabilities of different wordsappearing next to or near one another. For example, in some cases,natural language can either be statistically generated to completeusers' sentences by predicting highly probable repetitive and mundaneshort texts. In other cases, prior systems may generate text to resemblehuman-written texts, but with no effective control over the meaning ofthe text. That is, the text may appear structurally well-written, but toa reader would be understood as non-sensical, in whole or in part. Moreimportantly, prior systems do not allow a user to control the meaningconveyed by the generated text in such situations and, as a result,while a prior system may generate text that appears structurallywell-written, that text is unlikely to convey the meaning intended bythe user. This problem is heightened by the fact that a given word formcan possess multiple meanings. For example, the word “bass” can refer toa fish, a guitar, a type of singer, etc. Thus, the word itself is merelya surrogate of its actual meaning in a given context, which may bereferred to as the word's sense. In many cases, a context of surroundingtext may be needed to inform a word's sense. Prior systems tend togenerate text based on surface level statistics without accounting forcontext (e.g., user input indicating the desired meaning of the text;abstract semantic properties of the context, including representationsof the abstract meaning conveyed by the surface-level words comprisingthe context, such as senses; and context found subsequent to thelocation of text generation), such as the context offered by user inputor other available text in a document. As a result, while prior systemsmay generate text, without accounting for context or word sense, suchsystems may be useful only in generating simple, statistically formedword groups. There is no capability for generating more complex languagebased on the context dictated by surrounding text (e.g., text appearingbefore and/or after a text insertion point). And such systems fail toprovide the user with control relative to the text generated and,therefore, the user is unable to predictably control the meaning of thegenerated text or to refine the meaning of generated text with furtherinput to the system.

In still other cases, prior systems may generate language of apparentcomplexity, but such systems may be specially tailored to generatelanguage that conveys information from predefined datasets, forpredefined use cases, and/or in predefined ways. Certain systems mayalso automatically account for dictionary spellings of words and certaingrammar rules, but, in general, these systems are limited to operatingrelative to short text segments and without the benefit of contextualanalysis of surrounding text or of input provided by a user.

There is a significant need for automated natural language generationsystems capable of robust generation of text beyond the limitations ofprior systems. The disclosed embodiments provide methods and systems forgeneral-purpose controllable natural language generation. The disclosedembodiments allow for the automatic generation of unique naturallanguage that can express specific meaning, determined based oninteraction with users, based on analysis of existing text, etc. Thedisclosed embodiments can generate unique language, such as sentencesthat may have never been written before, the meaning of which can beeffectively controlled by users or by other parameters, for any desiredmeaning and context of the use of human language, with no need fortailored pre-configuration.

The disclosed embodiments also include semantically infused languagemodels. Such models may include a neural network-based language modelexplicitly trained to contain contextual relations between abstractsemantic features in text, in contrast with prior art, where models canonly be trained to learn contextual relations between surface-levelwords. For example, the disclosed systems may enable a model to learncontextual relations between words and word senses and between words andthe properties of the abstract concepts invoked by the text. To achievethis, the disclosed models may be trained to predict the semanticfeatures of masked tokens in text conditioned by their surroundingcontext.

As described in the sections below, the disclosed language generationsystems may provide a user with a significant level of control ingenerating language of an intended meaning that agrees with the contextof user input text and other available text. For example, in some cases,the disclosed systems may generate text output options as semanticparaphrase substitutions for input provided by the user. In other words,the text output options may be generated to convey the meaning,information, concepts, etc. of textual input provided to the system bythe user. Further, the disclosed systems, unlike prior systems, mayoffer a type of closed loop feedback where if text output optionsgenerated by the system do not quite match what the user intended, or ifthe user would like to supplement the generated text output options, theuser can modify the input to the system (e.g., adding words, removingcertain words, changing the order of words, etc.), and the system willautomatically generate one or more refined text output options based onthe modified input (and, in some cases, the context of text surroundinga document location where the generated text is to be inserted).

SUMMARY

Some of the presently disclosed embodiments may include a computerreadable medium including instructions that when executed by one or moreprocessing devices cause the one or more processing devices to perform amethod. The method may include: analyzing an electronic document text;identifying in the electronic document text a first drafted text elementto be modified; causing the first drafted text element to be highlightedto a user on a display; generating at least one text output option thatconveys a meaning associated with the first drafted text element butincludes one or more changes relative to the first drafted text element;and causing the at least one text output option to be shown to the uservia the display.

Consistent with the present embodiments, a system and method forreceiving user input of at least one word. The method may automaticallyconstruct at least one textual output option that differs from the userinput in at least one respect, expresses a meaning associated with theuser input, and agrees with a context associated with at least one textelement that is different from the user input. The method may also showat least one textual output option on a display.

Consistent with the present embodiments, a system and method forautomatically analyzing at least one text element and identifying one ormore contextual elements associated with at least one text element. Themethod may automatically construe at least one textual output optionthat differs from at least one text element in at least one respect,expresses a meaning associated with at least one text element, andagrees with at least one of the contextual elements identified relativeto the at least one text element. The method may also show at least onecontextual output option on a display.

Consistent with the present embodiments, a system and method forreceiving a request from the user to initiate a writing assistantapplication. The method may, in response to the request, cause a writingassistant workspace to be shown on a display. The method may receiveuser input, facilitated by the writing assistant workspace. The userinput may include at least one word that conveys at least one idea. Themethod may also automatically construct at least one textual outputoption that expresses at least one idea. The method may also beconfigured to show at least one textual output option in the writingassistant workspace on a display. The method may receive additional userinput, facilitated by the writing assistant workspace. The additionaluser input may include one or more additional words. The method may alsoupdate at least one textual output option based on the additional userinput.

Consistent with the present embodiments, a system and method forreceiving, from a user, an indication of a drafted text element in anelectronic document to be moved from a first location in the electronicdocument to a second location in the electronic document. The method mayalso move the drafted text element from the first location to the secondlocation in the electronic document. The method may generate at leastone text output option for insertion between the drafted text element atthe second location and an adjacent text element. The method may alsoshow at least one text output option on a display.

Consistent with the present embodiments, a system and method forreceiving, from the user, an indication of a text insertion location inan electronic document. The method may generate at least one text outputoption for insertion at the text insertion location in the electronicdocument. The text output option may link at least one aspect of a firsttext element that precedes the text insertion location with a secondtext element that follows the text insertion location. The method mayalso show at least one text output option on a display.

Consistent with the present embodiments, a system and method foridentifying in an electronic workspace a first text passage which mayinclude a first plurality of words. The method may identify in theelectronic workspace a second text passage which may include a secondplurality of words. The method may also analyze the first and secondtext passages to determine first information conveyed by the first textpassage and second information conveyed by the second text passage. Themethod may automatically generate a third text passage that conveys thefirst information associated with the first text passage and the secondinformation associated with the second passage. The third text passagemay be generated to include a first set of textual revisions relative tothe first text passage and a second set of textual revisions relative tothe second text passage. The method may also show the third text passageon a display.

Consistent with the present embodiments, a system and method foranalyzing an electronic document text. The method may identify in theelectronic document text a first drafted text element to be modified.The method may also cause the first drafted element to be highlighted toa user on a display. The method may generate at least one text outputoption that conveys a meaning associated with the first drafted textelement but may include one or more changes relative to the firstdrafted text element. The method may also show at least one text outputoption on a display.

Consistent with the present embodiments, a system and method forreceiving from a user an indication of a drafted text element in anelectronic document to be analyzed. The method may generate at least onetext output option that may convey a meaning associated with the draftedtext element but may include one or more changes relative to the draftedtext element. The method may also show at least one text output optionon a display.

Some embodiments may include a user input device. The user input devicemay include a writing assistant activation button configured to activatean automated writing assistant function, wherein the automated writingassistant function is configured to: receive user input including acollection of two or more words that convey at least one idea, whereinthe user input is received via a GUI element shown on a display inresponse to a user pressing the button; automatically construct at leasttwo complete sentence options that each express the at least one idea;and cause the at least two complete sentence options to be shown to theuser via the display.

BRIEF DESCRIPTION OF DRAWING(S)

FIG. 1 is a diagram illustrating an exemplary system environment inwhich the disclosed writing assistant may be used, consistent withdisclosed embodiments.

FIGS. 2 a-2 p show an embodiment of the writing assistant interface,according to exemplary disclosed embodiments.

FIGS. 3 a-3 i provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIGS. 4 a-4 g provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIGS. 5 a-5 f provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIGS. 6 a-6 o provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIGS. 7 a-7 f provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIGS. 8 a-8 d provide diagrammatic representations of a writingassistant interface, according to exemplary disclosed embodiments.

FIG. 9A illustrates an exemplary keyboard for use with the disclosedwriting assistant.

FIGS. 9B and 9C illustrate interface elements controllable using controlfeatures associated with the keyboard of FIG. 9A.

FIG. 10A provides a diagrammatic representation of a masked-wordprediction task.

FIG. 10B provides a diagrammatic representation of a masked-wordsupersense prediction task, according to exemplary disclosedembodiments.

FIGS. 11A and 11B provide diagrammatic visualizations of exemplarysupersense vectors learned by SenseBERT at pre-training.

FIG. 12A provides a diagrammatic representation of supersenseprobabilities assigned to a masked position within context.

FIG. 12B provides examples of SenseBERT's prediction on raw text.

DETAILED DESCRIPTION

The disclosed embodiments relate to a writing assistant system designedto generate useful, natural language output in a variety of situations.For many, tasks associated with writing can be arduous and slow. In manycases, writing may involve or require the generation of sentences and/ortext fragments that convey a particular meaning or concept, e.g., whencrafting text in support of a particular topic sentence, hypothesis, orconclusion; when developing bridging text (including transition phrases,sentences, or entire paragraphs) that link one section of a document toanother; when drafting text simply to convey various thoughts orinformation; or when generating any other forms of text.

Languages are complex, which can lead to added difficulties whenwriting. Each language has thousands of words, some of which may havesimilar meanings (e.g., synonyms) in certain contexts, subtledifferences in meaning in other contexts, or quite different meaningsdepending on the context in which the words are used. In some cases, aphrase may be used to convey an idea that may also be conveyed by asingle word, and vice versa. Sentence structure may also influence themeaning of text passages (e.g., order of clauses, proximity of amodifier relative to a subject, etc.). These are just a few of the manytypes of language variations that can lead to difficulties in developingwell-functioning, automatic natural language generator systems.

There is a significant need for systems having enhanced natural languagegeneration capabilities. For example, such systems may significantlyalleviate writing-related burdens experienced by users of traditionalsystems. The disclosed embodiments, in some cases, may receive inputfrom a user (e.g., a word, a phrase, or a grouping of words that mayconvey one or more ideas or bits of information) and may generatewell-formed text that conveys the meaning or information associated withthe user input. In view of the significant impact of context on themeaning of words or language, more generally, the disclosed systems seekto generate textual output that agrees with the context associated withother text, other user input, etc.

Such an operation may significantly increase the accuracy of generatedtext in conveying an intended meaning. For example, some statisticssuggest that up to 80% of global commerce is conducted using at leastsome English language communications for information transfer. But, onlyabout 20% of the world's population speaks English, and far fewer speakEnglish as a native language. This can lead to significant difficultiesor errors in conveying business information ranging from simple meetingdetails to complex agreement provisions or terms for negotiations, amongmany others. In some cases, the disclosed natural language generationsystems may generate one or more words, phrases, sentences, and/orparagraphs in response to input received from a user. For example, oneor more English language words entered into the writing assistant mayprompt the writing assistant system to generate one or more text outputsthat convey the idea and/or information associated with the user input.Such functionality may significantly ease the burden of non-nativeEnglish language speakers in generating business communications (or anyother communications) in the form of emails, term sheets, offer letters,supplier letters, contracts, among many others.

The disclosed writing assistant systems are also not limited tooperation solely in the English language. The writing assistant systemcan be trained relative to any language to either receive user input (orany type of text input) in any language and output text generated in thesame or different language. For example, in some cases, the disclosedwriting assistant systems may receive user input (or text input) in alanguage other than English and may output text options in English.

The ability of the presently disclosed systems to generate text output(e.g., well-formed text conveying information and/or one or more ideasthat may agree with a provided or determined context for the text) inresponse to input ranging from a single word, phrase, paragraph to alist of words, phrases, or paragraphs may also reduce the amount of timea user needs in drafting certain types of text. For example, a user ofthe writing assistant system may enter one or more key pieces ofinformation, and in response, the system may generate one or more textoutput options that convey the information. In one scenario, a user maystart an email with the words: meeting, my office, Tuesday at 11 am, andthe writing assistant system may return one or more text output options,such as “John, please stop by my office for a meeting on Tuesday at 11am,” among other varied options in text output structure, formality, orcontext. In embodiments where the system offers multiple text outputoptions, a user may select from among the options that best conveys theintended meeting. In some cases, the user can even select one of theoutput options that is closest to the intended meaning and have thewriting assistant generate one or more additional text output optionsthat are different from one another, but offer more refined optionsbased on the selected text from the initial list of output options. Instill other cases, the writing assistant system may update the outputtext options offered as a user enters additional input into the systemor as additional input otherwise becomes available.

In other disclosed embodiments, the writing assistant may generate oneor more words, phrases, or paragraphs, etc. that link together availabletext passages. For example, the writing assistant system may be providedwith a specific location in a preexisting text (e.g., using a cursor inan electronic document, etc.) and may offer linking text that bridgesbetween text appearing before and/or after the selected location. Suchbridging text may include one or more words, phrases, or paragraphs,etc. that convey concepts consistent with the surrounding text and agreewith one or more aspects of the context associated with the preexistingtext. The bridging text may be generated with or without prompts from auser (e.g., with or without the user providing the system with enteringadditional words conveying information and/or ideas for insertion into atext). Such approaches to text generation (and many other described morefully in the sections below) may enable users to more effectively andefficiently generate well-written text in less time than traditionaluser-generated writing tasks may require.

The disclosed writing assistant systems may also offer significantlyimproved text output options relative to those offered by traditionallanguage generation systems. For example, traditional systems tend to behighly rule-based and tied to probabilities relative to the appearanceof words in sentences, etc. As a result, such systems lack the abilityto provide text output options designed to account for availablecontext, either provided by a user or informed by preexisting text. Forexample, some systems can generate synonym suggestions for selectedwords, but such systems do not limit their output to synonyms that fitthe context of a document or surrounding text. Often, therefore, one ormore output options offered may be inappropriate or inconsistent withthe context of the user input and/or other text in a particulardocument.

Further shortcomings of prior word generators may arise from thestatistical way in which words are predicted and/or generated. Forexample, in these types of statistical model-based systems, one or morewords may be presented to a user as the user types into an interface.These words or phrases are typically presented to a user, for example,as an optional conclusion to a sentence being typed, and the fewrelatively simple words provided to the user generally are determined bythe system as a statistically most probable grouping of words thattypically follow the word or words entered by the user. Thesestatistical systems do not offer text generated as a replacement fortext input by the user that, for example, conveys ideas and/orinformation associated with the user's input. More importantly, suchsystems do not analyze context of the user input or other textassociated with a document in generating a text output. As a result, atext output generated from such a system may be inconsistent with thecontext of a document text, especially text other than text immediatelyentered by a user.

In some cases, prior word generators may provide lengthy outputs basedon one or more prompts. These systems produce text that may appearcomplex and well-structured. Indeed, some available systems can receivetext input prompts and generate multiple sentences or paragraphs inresponse. These systems, however, lack the ability to generate text thatagrees with or flows together with the information and context of textoutside of the prompts provided. As a result (and as one exampleshortcoming), the text outputs, which may have the structural appearanceof well-written text, typically read as nonsensical, randomly generatedstreams of sentences with little or no relationship to any surroundingtext. For example, unlike the presently disclosed writing assistant,prior systems lack an ability to generate textual outputs based on textthat follows a document location where a generated text output option isto be inserted. Such text generation systems often fall well short ofgenerating text useful to a user or that matches a user's intendedmeaning for a communication.

Further, while prior systems may include a graphical user interface(GUI), such prior interfaces are often limited in their functionalityand ability to interact with a user. The presently disclosed embodimentsare designed to offer a high level of interaction with users, dependenton a particular application. For example, in some examples, thepresently disclosed embodiments may provide multiple text output optionsin response to user input. The text output options, in some cases, mayconstitute complete sentences that incorporate and convey an idea,meaning, and/or information associated with the user input. Importantly,the text output options may also be generated by taking into account oneor more contextual elements associated with the user input and/or otherrelevant, preexisting text, such that the generated text output optionsagree contextually with the user input and/or preexisting text. The textoutput options may be updated as the user continues to provide inputsuch that the updated text output options offer refinements overinitially provided text output options in conveying the meaning, and/orinformation associated with the user input. To insert any of the offeredtext outputs into a document, for example, the user can make a selectionof one of the offered text outputs. Alternatively, the user can selectone of the text output options as a prompt for the writing assistantsystem to generate one or more additional text output options thatdiffer from one other, but may be more closely related to the selectedtext output options than to other non-selected text output options. Suchinteractive capabilities may significantly enhance a user experience andthe efficiency by which the user can generate well-written text thatconveys an intended meaning and agrees with the context of otherrelevant text.

The sections below describe in detail the functionality and features ofthe presently disclosed writing assistant systems. The sections alsoexplain in detail how such systems may be constructed to includeadvanced capabilities such as generating text output that both conveysconcepts and ideas included in user input (or other text) and agreeswith contextual elements of the user input and/or other text. In somecases, the disclosed writing assistant system may be based on trainedmachine learning language models trained to recognize complex contextualelements in text. For example, as alluded to above, such models may betrained, for example, using large corpuses of text, masking differentsegments of text (e.g., tokens), and one or more reward functions thatpenalize the system during training for generating text replacementsthat do not match the masked text and reward the system for generating atext replacement that matches the masked text. Such trained systems whenplaced into use, for example, may offer significantly improvedcapabilities for generating well-written text that conveys an intendedmeaning while agreeing with the context of surrounding text or otherrelevant text. Additional details regarding training of the network(s)associated with the disclosed writing assistant are discussed in moredetail in sections that follow.

Before turning to the details, it should be noted that the disclosedwriting assistant systems and their associated GUIs may be employedtogether with any type of computer-based technology. For example, suchsystems may be incorporated into word processing software, emaileditors, presentation software, or any other type of computerapplication in which text is involved. Additionally, the disclosedsystems may be operated on a PC, server, tablet, mobile device, laptop,heads up display unit, or any other type of hardware system capable ofexecuting an application including text-based functionality.

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings and disclosedherein. The systems and methods are described below in no particularorder and can be performed in any order and combination. Additionally,various embodiments of the disclosed writing assistant technology mayinclude some or all of the disclosed features and functionality in anycombination.

FIG. 1 is a schematic diagram of an exemplary system environment inwhich the disclosed writing assistant may be employed. For example,system 100 may include a plurality of client devices 110 operated byusers 120. System 100 may also include a network 130, server 140,internet resources ISO, cloud services 160, and databases 170. Thecomponents and arrangement of the components included in system 100 mayvary. Thus, system 100 may include any number or any combination of thesystem environment components shown or may include other components ordevices that perform or assist in the performance of the system ormethod consistent with the disclosed embodiments. The components andarrangements shown in FIG. 1 are not intended to limit the disclosedembodiments, as the components used to implement the disclosed processesand features may vary. Additionally, the disclosed writing assistantsystem may be implemented on any single component shown (e.g., a singlemobile device or single PC included in client devices 110) or may beimplemented in a network architecture (e.g., one or more features of thedisclosed writing assistant systems and methods being implemented on aserver 140, associated with one or more cloud services 160, etc. andhaving connectivity established with one or more client devices 110 vianetwork 130 (e.g., a WAN, LAN, Internet connection, etc.).

As shown in FIG. 1 , client devices 110 may include a variety ofdifferent types of devices, such as personal computers, mobile deviceslike smartphones and tablets, client terminals, supercomputers, etc.Client devices 110 may be connected to a network such as network 130. Insome cases, a user 120 may access the writing assistant and itsassociated functionality via the client device 110 which can display theuser interface of the writing assistant. For example, the writingassistant may be operated as a stand-alone application on a clientdevice 110, or the writing assistant may be incorporated into any textediting application that may be operated on a client device 110 (orother types of computing devices). In some cases, the writing assistantmay be incorporated with applications including, but not limited to,email editors, word processing programs, presentation applications,spreadsheet applications, PDF editors, etc.

Network 130, in some embodiments, may comprise one or moreinterconnected wired or wireless data networks that receive data fromone device (e.g., client devices 110) and send it to another device(e.g., servers 140). For example, network 130 may be implemented toinclude one or more Internet communication paths, a wired Wide AreaNetwork (WAN), a wired Local Area Network (LAN), a wireless LAN (e.g.,Bluetooth®, etc.), or the like. Each component in system 100 maycommunicate bidirectionally with other system 100 components eitherthrough network 130 or through one or more direct communication links(not shown).

As noted, the writing assistant may be implemented and run using avariety of different equipment, such as one or more servers, personalcomputers, mobile devices, supercomputers. mainframes, or the like,connected via various types of networks. In some embodiments, thewriting assistant may be configured to receive information from clientdevice 110, database 170, server 140, cloud service 160, and/or Internetsources 150 (among others) and send or return information to the same.The writing assistant can be incorporated into client devices 110 andrun locally or be run on a server 140 or from a cloud service 160accessed by the client device 110 via network 130.

As previously described, the writing assistant can be operated as astandalone application offering its own GUI or may be incorporated intoanother application (e.g. a parent application) and may offer one ormore GUI interface components to the parent application. For example,the writing assistant GUI (or parent application GUI supplemented withwriting assistant features) may provide a location to receive user input(e.g., at the cursor in editors, etc.). GUIs associated with thedisclosed writing assistant can also provide one or more windows orfields for receiving user input and one or more additional windows orfields for providing text output options in response to received userinput. The windows, fields, and/or functions of the writing assistantmay be selectively activated or deactivated. The user input may consistof words or text that can be extracted from a document or inputted bythe user using a keyboard or other appropriate input method, includingdictation by the user using voice recognition software. Multipleembodiments and examples of the writing assistant GUI along with variousfeatures associated with the disclosed writing assistant are discussedin the sections below.

In the disclosed embodiments, the writing assistant may allow users toexpress their ideas simply, for example, through simple naturallanguage, with no regard for correctness, grammar, style, clarity, etc.In response, the writing assistant may generate and provide to the userone or more suggestions (in some cases, several suggestions) for unique,well-written, and context-fitting texts that express the specifiedmeaning of the user input, and which may be inserted into the documentthat is being drafted. In contrast with existing grammar errorcorrection applications, for example, the disclosed writing assistantcan provide text options for the users ex-ante rather than correctingmistakes or making local suggestions ex-post. For example, whiledrafting initial text in a word processing user interface, a user maycall the writing assistant and write “lets make phone call, when is goodtime for you.” In response, the assistant would generate well-writtensentences that express the same meaning, such as “When are you free fora quick phone call,” “What times are you available for a phone call,” or“Can we schedule a phone call? What times are you available?”

FIGS. 2 a-2 p show a user interface that may be included with exemplaryembodiments of the disclosed writing assistant system. FIGS. 2 a-2 pshow an exemplary GUI 200 that may be associated with certain disclosedembodiments. In the example shown starting at FIG. 2 a , GUI 200 may beassociated with an email application and may include an email editor GUI205, which in turn, may include a workspace 210. In some cases, a usermay draft email text simply by entering text into workspace 210 withoutrelying upon features of the disclosed writing assistant. In some cases,however, entering text into workspace 210 may automatically triggercertain functionality associated with the disclosed writing assistantincluding, for example, the generation of text output options generatedby the writing assistant as possible replacements for the text enteredin workspace 210.

FIG. 2B illustrates an example in which the user enters text intoworkspace 210 prior to initiating the writing assistant. For example, asshown in FIG. 2B, the user has entered, “My name is Andrés López, I'mfrom ITG Group. I got your details from Jessica Abrahams,”. Inembodiments where the initiation of the writing assistant features arenot automatic, the user can select a GUI element, for example, toinitiate the functionality of the writing editor. Such GUI elements mayinclude, for example, menu items, virtual buttons, icons, etc. (notshown) that the user may select via a touchscreen, using a pointingdevice, or in any other suitable manner.

FIG. 2 c shows an example user input field 220 that may be presented onthe GUI in response to initiation of the writing assistant by the user.For example, a user can summon field 220 in the writing assistant, wherefield 220 is configured to receive text input from the user in the formof characters, words, sentence fragments, phrases, sentences,paragraphs, punctuation. etc. As shown in FIG. 2 d , a user can typeinput 225 into the field 220 (such as “and I understand from her”). Inresponse to user input provided to field 220 by the user, the writingassistant can generate various text output options as possiblereplacements for the input received at field 220.

For example, as shown in FIG. 2 e , in response to receiving the userinput, “and I understand from her,” the writing assistant can generateone or more text output options, such as text outputs 230 a-230 c, thatconvey a meaning or information associated with the user input, but mayuse different words relative to input 225.

The generated text output option(s) may be provided to the user in anysuitable format. In some cases, the generated text output options may beprovided to the user via output fields 230 a-230 c as shown in FIG. 2 e. Each of the provided text output options may appear in an individualfield 230 a, 230 b, or 230 c, for example. The individual fields may beindividually selectable and may provide the user with an option toselect from among the provided text output options for substitution intothe draft document in place of the text entered in field 220. Forexample, the selected text output option may be appended to text 215.

As shown in FIG. 2 e , the writing assistant can generate multipleoutput options that each differ from one another. Despite thedifferences, however, all convey the idea associated with the user input(e.g., that Jessica Abrahams conveyed information to the user, AndresLopez). Additionally, the text output options all agree with one or morecontextual aspects of the preexisting text (a partial sentence) inworkspace 210. For example, the phrase “and I understood from her” issimilar to the input, but changes “understand” to “understood” forconsistency with the tense of the preexisting text (i.e., the word “got”appears in the past tense in text 215). This phrase also indicates thatthe writing assistant detected that Jessica Abrahams is a female eitherbased on the user input, on the preexisting text 215, or a combinationof both. As a result, the writing assistant substituted the pronoun“her” for the name of the person that gave Martin's details to AndresLopez. Option 2 (i.e., “and she told me”), while including differentwords from Option 1, conveys a similar meaning and replaces JessicaAbrahams with the pronoun “she” to indicate a recognition that Ms.Abrahams is finale in agreement with the preexisting text. Option 3includes yet another organization of words conveying a similar meaningas the user input and also showing agreement with the context of thepreexisting text by substituting Jessica Abrahams with the pronoun“she.” Options 2 and 3 also use the past tense in agreement with thepreexisting text, despite the use of the present tense in the userinput. Notably, while option 1 (field 230 a) uses the phrase “understoodfrom her,” which is similar to the words appearing the user input,options 2 (field 230 b) and 3 (field 230 c) include very differentwords, but still convey a similar meaning as the user input. That is,option 2 includes the phrase “she told me,” and option 3 includes thephrase “she said to me,” which both indicate that Jessica Abrahamsconveyed information to Andrés López. While the phrases in options 2 and3 are not synonymous with the phrase in option 1 or with the user input,they all convey similar meanings, especially when considering thatspeaking is a primary form of communication and one often associatedwith a characterization of whether a recipient of spoken wordsunderstands what the words of the speaker conveys to the recipient.

In some embodiments, the text output options are not static, but rather,can be updated as a user continues to provide input to field 220, forexample. In FIG. 2 f , the user types updated input 235 that adds thephrase “you want to hear more on what we do” to the originally entereduser input, “and I understand from her.” In response to receiving theupdated user input, as shown in FIG. 2 g , the writing assistant willgenerate a set of updated text output options 240 a-240 c, which may ormay not include the originally generated text output options. In theexample shown in FIG. 2 g , the writing assistant generates the outputoption “and I understood from her that you would love to know more aboutwhat we do in Greece” (field 240 a). In addition to changing“understand” to “understood” for consistency with the tense of thepreexisting text, the writing assistant changes “you want to hear more”to “you would love to know more,” which indicates that the writingassistant detected the context of the additional text and suggested,among several changes, using “love to know” instead of “want to hear” inthis context. This is an example of the writing assistant's ability touse a word or phrase that conveys a similar meaning in the particularcontext of the user input despite the words/phrases used in the textoutput option not being recognized synonyms for the words/phrases of theuser input. In some cases, however, the writing assistant can also offertext output options that include words that are recognized as synonymsto words of the user input (e.g., word pairs that may be found in astandard thesaurus, such as the Historical Thesaurus of the OxfordEnglish Dictionary).

Returning to FIG. 2 g , option 2 (i.e., “and she told me that you wereinterested in our business in Greece”) 240 b also conveys a similarmeaning to the user input, but uses a different phrase (i.e., “that youwere interested in”) from the input or the other text output options.Option 3 (i.e., “and she told me about your interest in the opportunityin Greece”) 240 c, again, conveys a similar meaning but with a differentphrase (i.e., “about your interest in”).

Notably, all three options reference the detail that the activities areoccurring in Greece, despite there being no reference to Greece ineither the user input in field 220 or in the preexisting text 215. Forexample, the writing assistant. as evidenced by the text output options,was able to determine that ITG Group is a real estate group operating inGreece. The writing assistant is able to pull contextual information notonly from the words of the user input and/or the words of thepreexisting text, but also from other available sources of information(e.g., Internet-accessible databases, among others). The feature isdiscussed in depth later in this disclosure.

Once the text output options provide the user with suitable text, theuser can select one of the text options. For example, a user may selecttext output 240 c, as shown in FIG. 2 h . In response, as shown in FIG.2 i , the writing assistant can insert the user-selected text outputoption 240 c into the workspace 210 with the initial text 215, creatinga coherent and context fitting paragraph (e.g., inserted text 245).

The drafting process can continue with the user entering additional userinput (e.g., via a second field 250, which may be a newly displayedfield or a continuation of user input field 220), as shown in FIG. 2 j .Similar to the description above, the writing assistant can use theinserted text 245 (e.g., preexisting text) and additional input includedin field 250 to generate additional context-fitting text output options.As shown in FIG. 2 j , after the inserted text 245 is inserted intoworkspace 210, the user can summon a second field 250 (e.g. a window,text box, etc.) that may be visible when the writing assistant is activeand not visible when the writing assistant is inactive. As noted, insome cases, field 250 may be the same as field 220. Or, in some cases,field 250 may appear if the user hovers over a predetermined region ofthe GUI in order to activate field 220/250. In the embodiment of FIG. 2j , the user may provide second input 255 into second field 250. Theuser input may include a collection of words (e.g., one or more words,phrases, etc.) that convey at least one idea or piece of information.The collection of words may include a word, a sentence fragment, acomplete sentence, or clauses that can each convey a unique idea. Thecollection of words may also identify a subject and at least oneattribute of the subject, for example, a name of person, a name of anorganization, a time associated with an event, a name of a place, or aplace associated with an event. The subject itself may identify anentity that is a person, a place, a thing, an organization, acorporation, an event, or some other appropriate identifier.

In response to input received from the user (e.g., text entered intosecond field 250), the writing assistant may generate any number of textoutput options and may provide those text output options in one or moresecond text output fields 260 a-260 c, as shown in FIG. 2 k . In somecases, the assistant may generate one text output option in response tothe user input. In other cases, two or more text output options may beprovided, where the two or more text output options each express atleast one idea and where the text output options differ from one anotherin at least one respect. Offering multiple text output options mayenable the user to select the generated text output option that mostclosely conveys an intended idea or that most closely fits with thecontext of the document.

As shown in FIG. 2 j , a user may begin to type a second input 255 in asecond field 250 (“Lets make a phone call and talk”). The writingassistant, as shown in FIG. 2 k , may generate second text outputs 260a-260 c that, like the text outputs described above, are intended toconvey the same meaning as the user input, but with well-written,context-fitting text. But, instead of choosing a second text output, auser may, as shown in FIG. 2 l , prompt the generation/display of anadditional field 265. As shown in FIG. 2 m , a user could enteradditional input 270 in the additional field 265 (“When it is possiblefor you?”). In response, the writing assistant may generate updated textoutput options 275 a-275 c (FIG. 2 n ) that take into account theinformation from inserted, preexisting text 245, second input 255, andthe additional input 270. As shown in FIG. 2 o , the user can select anyof the generated text output options included in fields 275 a-c. Itshould be noted that text output options included in fields 275 a-c mayhave been generated as the user began entering text input into field250, and the writing assistant may have updated the text output optionsone, two, or more times as the user continued entering text into field250 and further as the user entered text into field 265.

In the example shown, the user selects text output option 275 b (FIG. 2o ), and as shown in FIG. 2 p , the writing assistant may automaticallyinsert the selected updated text output 275 b into the workspace 210,creating a well-written, grammatically correct email (i.e., updatedinserted text 280). In some cases, the use of two different input fields250 and 265 may indicate to the writing assistant that two differentsentences are intended, and, as a result, the text output options may bepresented with multiple sentences (e.g., each corresponding to theconcepts conveyed in a separate user input field).

In addition to text output options that include phrases or sentencefragments, as shown in FIG. 2 e , the disclosed writing assistant systemcan provide text output options in various other forms. In some cases,based on the received user input, the writing assistant canautomatically construct multiple text output options that each expressat least one idea associated with the received user input and where thetext output options are provided in the form of complete sentences,multiple complete sentences, full paragraphs, multiple paragraphs, etc.For example, as shown in FIGS. 3 a-3 i , in response to received userinput, the disclosed writing assistant may generate one or more textoutput options in the form of complete sentences that may convey an ideaor information attributed to the received user input. The completesentence options, as with other text output options of the disclosedwriting assistant, may also agree with one or more contextual aspects ofthe received user input or other relevant text (e.g., preexisting textin a document being drafted by the user). For example, GUI 300 may beassociated with an email editor 305 (or stand-alone writing assistantapplication or any other computer application that allows for textentry) and may include a workspace 310. As shown in FIG. 3 b , a usercan summon a field 315 in the writing assistant (e.g., by initiatingtyping in workspace 310, positioning a cursor relative to workspace 310,hovering a cursor over a designated area associated with the GUI,selecting a menu item associated with the writing assistant, clicking ona virtual button to initiate the writing assistant, or any othersuitable technique for initiating the writing assistant application).Similar to the example above, the writing assistant may functionrelative to text the user enters directly into workspace 310 and/or mayfunction in response to text entered by the user into input field 315,as shown in FIG. 3 b . As shown in FIG. 3 c , a user can enter textinput 320 into field 315. Text input 320, provided in field 315, forexample, may include one or more words, phrases, sentence fragments,sentences, clauses etc. with which the user may use to convey ideas,information, and/or to indicate context, etc. In the example shown inFIG. 3 c , text input 320 includes the phrases. “building delays inDenver; lots of design changes.” As shown in FIG. 3 d , the writingassistant create full-sentence text outputs options 325 a and 325 bbased on these inputted phrases included in text input 320. While twotext output options are shown in FIG. 3 d , the disclosed writingassistant may generate more or fewer text output options. As shown inFIG. 3 e , the user can select from among the generated text outputoption. In this case, the user selects the text output option 325 a,which reads, “Our building project in Denver has been slowedsignificantly by the need for unexpected design changes.” Next, as shownin FIG. 3 f , the writing assistant can insert the selected text outputoption into workspace 310 as inserted text 330.

This drafting process, augmented by the writing assistant applicationmay continue as long as the user has additional concepts or informationto convey. For example, as shown in FIG. 3 g , the writing assistant GUI300 may include a field 335 for receiving user input. As in the exampledescribed above, field 335 may constitute a newly generated field (e.g.,a second field initiated by activation of a writing assistant controlelement). In other cases, however, field 335 may be the same as field315, once emptied of any previous user input, such as input 320. In somecases, selection by the user of a generated text output option (e.g.,one of text output options 325 a or 325 b) may automatically result infield 315, 335, etc. being cleared of text input by the writingassistant application in order to prepare for the entry of additionaluser input into field 315, 335, etc.

To generate a second sentence for the document, the user can provideinput to field 335, and the writing assistant can generate text outputoptions in response. As shown in FIG. 3 h , the user may provide to thesystem, as input 340, the group of words: “meeting Tuesday 2 pm costoverruns.” In response, the writing assistant may populate one or more(e.g., two or more) text output fields 345 a and 345 b (which may takethe form of windows, text boxes, etc.) with the text output optionsgenerated by the writing assistant based on the user input 340. Thewriting assistant may also base the text output options upon textalready existing in the document workspace 310. For example, as shown inFIG. 3 i , the document being drafted includes inserted text 320 (e.g.,text inserted into workspace 310 by the user's previous selection of thetext appearing in field 325 a) that reads, “Our building project inDenver has been slowed significantly by the need for unexpected designchanges.”

The writing assistant can use both the user input 340 and the insertedtext 330 in generating the text output options provided in fields 345 aand 345 b. In some cases, contextual information may be determined bythe writing assistant analyzing inserted text 330 and/or user input 340.The writing assistant may also generate the text output options toconvey the same or similar ideas or information detected as included inuser input 340, even where user input 340 does not include completesentences. That is, despite not representing a complete or grammaticallycorrect sentence or grammatically correct sentence fragment, the writingassistant can determine an idea and/or information associated with theuser input 340 (in this case, that the user would like to request ameeting on Tuesday at 2 pm to discuss cost overruns associated with thebuilding project). In response, the writing assistant can automaticallygenerate one, two, or more complete sentence options that convey themeaning and/or information associated with the user input 340. Forexample, as shown in FIG. 3 i , a first complete sentence options shownin field 345 a may read, “Can we schedule a meeting on Tuesday at twoo'clock pm Mountain time to discuss cost overruns?” Another text outputoption, shown in field 345 b may read, “We need to talk about costoverruns. Are you free at 2 pm Mountain time?” Notably, both text outputoptions convey the idea and information that the user is interested in ameeting at 2 pm on Tuesday regarding cost overruns. Notably, as theexample of FIG. 3 i shows, the writing assistant text output options maybe complete sentences, despite the user input constituting less thancomplete sentences. Further, the text output options may include two ormore complete sentence options even where the user input includes lessthan a single complete sentence.

As in the previous examples, the writing assistant can also generate thetext output options included in fields 345 a and 345 b such that theyagree with contextual aspects of other relevant text, such as the userinput 340 and/or the inserted text 330. For example, both text outputoptions shown in FIG. 3 i , include a clarification that the timerequested for the meeting is relative to the Mountain time zone. Thesystem may include such a clarification, for example, by recognizingthat the preexisting sentence related to a building project in Denver,which the system automatically recognized/determined as located in theMountain time zone of the United States.

The text output options generated by the disclosed writing assistantsystems may convey any conceivable ideas or information that may beincluded in or associated with a user input. For example, in some commonexamples, the expressed ideas of the text output options may include,but are not limited to, a time for a meeting, a request for a meeting, apurchase request, or various ideas/information conveyed by one or moreentered clauses (e.g., when a delivery is expected to arrive, when alast meeting occurred, an indicator of an attribute associated withcertain goods or services, among hundreds of thousands of other types ofclauses).

The text options automatically generated by the writing assistant may besimilar to the received user input (e.g., compare the input in field 335of FIG. 3 i to the first text output option provided in output field 345a). In other cases, however, the generated text output options, whetherrepresenting complete sentences or not, can differ significantly fromthe user input. In fact, in some cases, the text output optionsgenerated by the writing assistant may include none of the words fromthe user input and, instead, may convey the ideas, meaning, and/orinformation associated with the user input using entirely differentwords than those included in the user input.

The text output options automatically generated by the writing assistantmay differ from the user input in various other ways. For example, thetext output options may include a re-ordering of the subject, verb,adjectives, pronouns, or any other attributes from a collection of wordsassociated with or included in the user input. And, as described above,the writing assistant can extract at least one higher-level attributeassociated a subject associated with the user input. For example, suchhigher-level attributes associated with the subject may include, but arenot limited to, a gender of the subject, a relation of the subject tothe user, an education level indicator of the subject, or a relation ofthe subject to another entity. An example of this type of extraction ofhigher level attributes associated with the subject of a user input isshown in FIG. 1 b where the writing assistant automatically determinedthat Jennifer Abrahams likely identifies as a female and, therefore,replaced her name in the text output options with the pronouns “her” or“she.” This is a subtle, but especially powerful feature, as the textoutput options provided in FIG. 1 b all sound more natural to a readerthan if the name “Jessica Abrahams” was repeated again in the samesentence.

It should be noted that while the embodiments of FIG. 2 and FIG. 3include fields (e.g., field 315 in FIG. 3 b ) for entering user input,the disclosed embodiments of the writing assistant are not limited toreceiving user input via such text entry fields. Rather in some cases,and as noted above, the writing assistant may monitor text entered inworkspace 210/310. for example, and may generate text output optionsbased on text that a user may enter directly into the workspace. Forexample, in some cases, the writing assistant may focus on subsegmentsof text provided in workspace 210/310 and use those subsegments as theuser input for generated text output options. Such text subsegments mayinclude, for example, text that a user inputs in workspace 210/310 aftera preceding period or other sentence ending punctuation. In other words,for each new sentence that a user wishes to include in a new document,the user may enter one or more words, sentence fragments, group ofwords, etc. that convey an idea, meaning, or piece of information. Inresponse to the enter words, etc., the writing assistant can providetext output options (e.g., in the form of complete sentences, etc.) thatconvey a meaning, idea, and/or information of the user input and thatagree with preexisting text. The user can select from among the providedoptions such that the selected text output option is appended to thedocument in place of the current user input. The user then moves on toconstructing a new sentence by providing another series of words, etc.that trigger the writing assistant to generate another series of textoutput options associated with the newly received user input (e.g.,newly entered after a period or other sentence-ending punctuation, aftera carriage return, etc.). In addition to supplying user input via typedtext, any other suitable input methodology may be employed for providinguser input. In some cases, for example, user input may be provided viavoice recognition applications.

When automatically constructing the complete sentence options (or othertypes of text output options), the writing assistant can usepredetermined style parameter values or selected user-selected styleparameter values n constructing the text output options. These styleparameter values may be used to generate an initial set of text outputoptions. Alternatively, or additionally, the writing assistant may usethe style parameter values to further refine certain text output options(e.g., options selected or indicated by a user).

FIGS. 4 a-4 g illustrate another example of possible interaction betweenthe writing assistant and a user during generation of text for adocument. Again, an email editor 405 is shown as the environment inwhich the writing assistant is employed, but any other text-relatedcomputer application may also be used. In the example, of FIG. 4 a , theuser can summon a field 420 in a workspace 410 using any suitabletechnique, such as those described above. In some cases, workspace 410may include preexisting text 415 already entered by the user (or whichmay already appear as part of a preexisting document, such as a Wordfile, etc.). As show in FIG. 4 b , the user can enter text input 425(“Thanks for the meeting with Michael”) into user input field 420. Inresponse, similar to the examples described above, the writing assistantcan automatically generate text output options 430 a-430 c.

In this example, the text output options may be included together withvarious control elements, such as icons 435 and/or icons 436 in GUI 400.Such control elements may be used by the user to control variousinteractions with the writing assistant. For example, in order to selectone of the text output options and to cause the selected text outputoption to be inserted into the workspace (as described in the examplesabove), the user may click on or otherwise select an icon 436 thatcorresponds with the desired text output options. In response, thewriting assistant may cause the selected text output option to beinserted into the workspace.

Other control elements may be included as well. For example, as shown inFIG. 4 c , the user can select any of the icons 435 to initiate one ormore functions associated with the selected icon. In the example shown,a user may select icon 435 a (denoted by gray highlighting over icon 435a) that corresponds with a particular text output option 430 a. Inresponse to selection of icon 435 a, and as shown in FIG. 4 d , thewriting assistant GUI 400 can display another window (e.g., a styleparameter control window) that identifies style parameters 440 (e.g.,parameters 440 a-d) for which values may be selected by the user. Thevalues for the predetermined style parameters (which, in some cases, canbe built into the writing assistant or which may be user-selectable) mayspecify a level of formality, conciseness, emotion, politeness, or alevel associated with any other parameter type that may be relevant tothe document. For example, in some cases, the user may control thelength of the text output options (e.g., complete sentences orotherwise) using the conciseness control. Alternatively or additionally,a text output option length selector (not shown) may be included toenable a user to specify a desired maximum length (e.g., 8 words, 12words, 20 words. etc.) for the generated text output options or tospecify a desired length range for the generated text output options(e.g., between 5-10 words, 11-20 words, etc.).

As shown in FIG. 4 e , the user can edit the level of the styleparameters using the displayed toggles 480 (or any other suitable GUIcontrol elements) or by manually typing entering the adjustment herselfvia the modifier windows 442. For example, as shown in FIG. 4 e , theuser has adjusted the level of formality 440 b down to “−1” (e.g., to alower level of formality using toggles 480 or modifier window 442). Thischange may cause the writing assistant to automatically update the textassociated with selected text output option 430 a according to thechange in parameter value. For example, as shown in FIGS. 4 d and 4 e ,the reduction in level of formality may cause the writing assistant tochange the selected text output option (“I wanted to thank you forarranging the meeting with Michael”) to the adjusted text 485 (“Thanksfor putting together the meeting with Michael”).

The adjusted text 485 is less formal than the original selected text 430a. For example, as FIGS. 4 d and 4 e show, in response to the change informality level, the writing assistant makes several changes, such aschanging “thank you” to “thanks” and “arranging the meeting” to “puttingtogether the meeting” to lessen the level of formality.

The user may continue to adjust the level of formality up or down, andin response, the writing assistant may continue to generate updated textfor the text output option to reflect the user's change in formalitylevel. Of course, other available parameter values may also be changed.In the example shown in FIG. 4 d , the user can make adjustments to thepoliteness, emotion, and conciseness parameter levels (e.g., usingtoggles 480. And in response to a change in value of any of theavailable parameters, the writing assistant may generate updated textfor the text output option to reflect the user's changes.

As shown in FIG. 4 f , once the user is satisfied with the adjusted text485, the user can select the adjusted/refined text output by selectingthe user acceptance icon 445. As shown in FIG. 4 g , the writingassistant can automatically insert the adjusted/refined text into thedocument or email workspace 410 as inserted text 450. This feature isnot limited to style parameters such as politeness, formality, etc. Theuser may also specify other aspects of the text output options, such asa text output length, as described above. Further, a user-specifiedlength for the text output options can be expressed numerically, asdescribed above, or may be expressed more generally as short, medium, orlong. For example, in the parameter level control window, the writingassistant may show the options short, medium, and long on the displayfor the user to choose. In another example, the writing assistant mayinclude toggles similar to those in FIG. 4 f that may allow the user toincrementally increase or decrease the number of words provided in atext output option (including a selected text output option, such astext option 430 a. For example, selected text output option is 11 wordslong, but if a user wished to shorten or limit the length of the textoutput option to 10 words, the user could enter “10” in a length styleparameter modifier input field (by toggle, typing, voice recognition.etc.). In response, the writing assistant would automatically refine theselected text output option to adhere to the user-imposed lengthlimitation. For example, the writing assistant could change the selectedtext output option 430 a to “Thank you very much for arranging themeeting with Michael.” to convey the original meaning of text outputoption, but within the 10-word limit.

As described above, the writing assistant can automatically constructtextual output options that differ from the user input in at least onerespect, express a meaning, idea, or information associated with theuser input, and also agree with a context associated with text elementseither found in the user input or within text (e.g., preexisting text ina document workspace) that is different from the user input. Contextualagreement may have various meanings. In some cases, however, anagreement between two or more text elements may refer to grammaticalagreement (e.g., the insertion of the generated text output option doesnot result in a grammar error relative to the preexisting text). Inother cases, agreement between text elements may be achieved by thegenerated text output options being generated to include in the same orsimilar style as the text around it (e.g., preexisting text in adocument workspace). Another contextual agreement may exist where agenerated text output option connects coherently to the text around itonce inserted into a document workspace. This form of agreement mayinclude, but is not limited to, the generated text being related to thesame general subject as the context and/or events or facts referenced ina generated text output options being consistent with events or factsreferenced by preexisting text in a document workspace, for example. Theconsistency may be relative to a relationship (e.g., temporal, causal.teleological, explanatory, etc.) existing between generated text outputoptions and preexisting text or user input. Contextual agreement mayalso exist where facts implied by generated text output options areconsistent with facts implied by the preexisting text; where temporaland causal relations between facts or events referenced in generatedtext output options and in the preexisting text are not implausible inlight of real-world constraints (e.g., a person can't perform an actionafter he dies, an event cannot start after it ends, a person cannot belocated in two different locations at the same time, etc.). A possibletest of contextual agreement between preexisting text and text outputoptions generated by the writing assistant may include whether more thanseventy percent of human evaluators are not able to discern that agenerated text output option, once inserted into the preexisting text,was generated by a machine rather than by a human. In addition tocontrolling text style using style control parameters, the disclosedembodiments of the writing assistant may also be configured to apply adefault style that is predetermined or learned based on usage. Forexample, the writing assistant may learn the personal style of the useror the style of a particular organization, in different contexts (e.g.,based on internal business documents, external business email. personalemail, etc.). In this way, the writing assistant may generate suggestedtext output options in a style that resembles that personal ororganizational style in the specific context of the document.

Further, in addition to enabling the modification of individual textoutput options, the writing assistant may also be configured to enableusers to modify the desired style of entire document. In response tosuch a selected action, the writing assistant may automatically rephrasethe existing document text and all text generations in that documentgoing forward in accordance with one or more selected style parametervalues to be globally applied. Similar to other described examples, suchstyle parameters may include formality, conciseness, politeness,emotion, sentence length, etc.

Additionally or alternatively, the writing assistant may enable users toselect any piece of text, e.g., in the document being written or inanother source, and choose to copy that text's style. For example, thewriting assistant may detect at least one style attribute (politeness,emotion, formality, etc.) associated with the selected text and then mayuse or apply such a style attribute in modifying other text in thedocument. For example, a user may select any piece of text in thedocument and choose to ‘paste’ the copied style attribute. The assistantwill then automatically rephrase the target text such that its styleresembles that of the source text or the assistant may offer one or moretext output options that rephrase one or more segments of the targettext in the style of the source text.

Disclosed embodiments of the writing assistant are not limited to thegeneration of text options based in response received text input from auser. For example, in some embodiments, various text segments (one ormore words, sentence fragments, phrases, sentences, paragraphs, etc.)may be identified in an existing document (e.g., either automatically orbased on user control), and in response, the writing assistant maygenerate one or more text output options relative to the identified textsegments. FIGS. 5 a-5 f show one example of such functionality providedby the disclosed writing assistant applications. FIG. 5 a shows anexemplary email editor 505 including a workspace 510 (although any othertype of text-based computer application may be used in conjunction withthe disclosed writing editor or the writing editor may be embodied as astand-alone application). As shown in FIG. 5 a , the email documentincludes preexisting text 515.

The presently disclosed embodiments of the writing assistant mayautomatically analyze preexisting text 515 and identify text elementsfor which the writing assistant may offer one or more text outputoptions as alternatives. For example, as shown in FIG. 5 b , the writingassistant may automatically analyze text 515 and identify text elements,such as highlighted text 520, for which the writing assistant may offeralternative text output suggestions. Such automatic analysis may beinitiated as part of a routine called by the user so that the writingassistant scans the text and offers suggestions for fixes (e.g., two ormore alternative text options for the user to consider as alternativesto the highlighted text 520).

It should be noted that there may be additional techniques for causingthe writing assistant to analyze text within a preexisting document andoffer suggested alternative text relative to identified text. Forexample, such functionality may be provided automatically as a userenters text into a workspace. That is, if a user enters a text elementinto a workspace that the writing assistant determines may be improved,the writing assistant may alert the user by highlighting the enteredtext or by any other suitable technique. In some cases, the writingassistant may automatically generate one or more alternative text outputoptions for the user to consider. In other cases, the user may berequired to confirm an interest in viewing alternative text outputoptions for entered text by, for example, selecting a GUI interfaceelement, etc. The writing assistant's analysis of entered text elementsmay be triggered by any suitable action, such as entry by the user of aperiod or other sentence-ending punctuation, entry of a carriage return,etc. Additionally, a user may select a GUI icon, menu entry, etc. toinitiate review of drafted text by the writing assistant. Such a GUIicon may include any suitable type of virtual button, etc. Menu entriesmay be selected, for example, from a drop-down menu (e.g., a Reviewtab). The automatic analysis of preexisting text elements by the writingassistant may also be initiated by the user manually highlightingcertain text elements, which may trigger the analysis by the writingassistant and the generation of text output options. In some cases, theuser may initiate review of a text element by the writing assistant byhighlighting a certain text element and then clicking on or otherwiseselecting one or more GUI control elements, icons, buttons, or menuitems.

Returning to the example associated with FIGS. 5 a-f , as shown in FIG.5 c , the assistant may automatically analyze the highlighted text 520in response to any of the triggers described above or in response to anyother suitable trigger for the review functionality. In some cases, anindicator 525 (e.g., a spinning wheel, hourglass, etc.) may indicatethat the writing assistant is analyzing the highlighted text 520together with text 515 (e.g., to determine context within which thegenerated text output options are to fit). As a result of the automaticanalysis, the writing assistant can automatically generate text outputoptions, such as text output options 530 a-530 c that the user mayconsider as possible replacements for highlighted text 520. Aspreviously described, each of the generated text output options maydiffer from the text elements included in the highlighted text 520 in atleast one respect, but may express a meaning associated with the textelements, while agreeing with contextual elements associated with text515 and/or highlighted text 520.

Moving to FIG. 5 d , the writing assistant has generated three textoutput options 530 a-c. Each conveys a meaning similar to meaningassociated with the highlighted text 520 (“it will probably not be muchbetter than ALP2”). Notably, however, as the generated text outputoptions suggest, the writing assistant automatically determined that theterm “It” in the highlighted text 520 may be unclear. In response, eachof the generated text output options rectifies this potential confusionby clarifying that the drafter is likely referring to an expectedimprovement over the ALP2 system. Additionally, text output options usethe pronoun “We,” which agrees with the context of the preexisting text515, which includes words such as “us” and “our,” which suggest thedrafter is referring to a group of people to which the drafter maybelong. Additionally, each of the text output options further agreeswith the context of the preexisting text 515 at least by offering aprelude of the “thoughts” that the drafter and the group to which thedrafter belongs expect to later articulate to Adam Rosenthal during theproposed conversation (i.e., that the improvement over ALP2 is notexpected to be significant or substantial).

As shown in FIG. 5 e , if any of the generated text output optionsbetter fits the meaning that the drafter intended to convey with thehighlighted text (or that the user simply prefers over the highlightedtext), the user can select one of the generated text output options as areplacement for the highlighted text Any of the techniques and functionsdescribed above (e.g., techniques by which a selected text output optionmay be indicated, techniques by which a user may cause the writingassistant to further refine any of the generated text options, controlof style parameters, etc.) may be incorporated into the embodimentrepresented by FIGS. 5 a -f.

As shown in FIG. 5 e , the user has selected text output option 530 b.In response, the writing assistant can automatically substitute theselected text output option 530 b for the highlighted text to provideinserted text 535 in workspace 510, as shown in FIG. 5 f.

It is important to appreciate that the writing assistant can alsoanalyze text in a document based on where that text is located in thedocument and in relation to other pre-existing text 515. For example, insome cases. highlighted text (or text for which the writing assistant asidentified for potential substitution with a text output option) mayappear at the beginning, middle, or end of a paragraph. In some cases,the highlighted text may appear in the middle of a sentence. In eachcase, the writing assistant may generate any of the text output optionsbased on where the highlighted text (or text to be replaced) appears inthe document. Sentences near the beginning of a paragraph may be framedas a topic sentence and/or may be more likely to identify subjects byname without use of pronouns. Sentences near the end of a paragraph maybe framed as a conclusion, and sentences to appear in the middle of aparagraph may be framed as supporting of the a topic sentence and/orconclusion that may be included in the paragraph. These are just someexamples of how the writing assistant may generate text output optionsbased on the intended location in a document for the generated textoutput options.

In some cases, the writing assistant may generate text output optionsnot as substitutes for text that already appears in a document, butrather as linking or bridging text. For example, a user may place acurser, for example, at a location in a document where the user wouldlike the writing assistant to generate and insert text. In some cases,the user may place the cursor in the middle of a sentence. In othercases, the user may place the cursor between paragraphs. at thebeginning of the document text, at the end of the document text, etc. Inresponse, the writing assistant may generate one or more text outputoptions for insertion at the cursor location. In such cases, rather thanbasing the text output options on highlighted text or user-entered textin a user input field, for example, the writing assistant may generatean original text output based on text that may precede or follow thecursor. For example, the writing assistant may draw subjects andinformation from the surrounding text and formulate linking or bridgingtext objects that synthesize those subjects and information into textthat expands on or further modifies the existing text. Text appearingcloser in proximity to the cursor location may have a stronger effect onthe words or language elements that the writing assistant automaticallyselects for inclusion into the generated text output options. As aresult, the generated text output options may offer text that flows withand connects naturally with the surrounding text, especially the text inclose proximity to the insertion location.

Again, any of the functionality described elsewhere may be incorporatedinto or used with this particular example. For example, in some cases,generation of linking text by the writing assistant may be controlledwith user-selected parameter values, similar to those shown in FIGS. 4a-4 f . For example, if the user places a cursor at a certain locationin the workspace, the user may be able to select or indicate the type oftext to be inserted at the cursor location (e.g., a sentence, aparagraph, a figure caption, etc.). All of the other previouslydescribed parameter value options, among others, may also be availableto the user in an embodiment in which the writing assistantautomatically generates text based on a selected location in a document.

In another exemplary embodiment of the system, consistent with disclosedembodiments, the writing assistant can construct text output optionsbased, at least in part, by accessing and relying upon sources ofexternal information (e.g., outside of the document, outside of what theuser inputs, outside of or remotely located relative to a device, suchas a PC or mobile device, on which the writing assistant is implemented,etc.). As shown in FIG. 1 . for example, the system may access internetsources 150, databases 170, or any other remotely located devices ordata repositories, etc. via network 130.

In some cases, information retrieved or accessed from the remotelylocated devices or databases, for example, may be used by the writingassistant in various ways. In some instances, the writing assistant mayuse such information to verify aspects of preexisting text in a documentand/or the generated text output options. For example, the writingassistant may use the externally available information to verify thatthe generated text output options do not contradict the externallyavailable information. In other words, the writing assistant can comparefacts to be included in generated sentences/text output options toverify that they are aligned with information from one or more externalknowledge bases. As one example, an agent could be in Paris and Franceat the same time but not in Paris and England at the same time. In thisexample, the writing assistant may receive the location “Paris” from theuser. The writing assistant can access the Internet and through searchengines, social media, and/or some other type of data mining, and byusing other contextual clues in the document (e.g., a company namereferenced in an email, etc.), the writing assistant may automaticallydetermine that Paris, as referenced by the user, must be a location andthat it can be in Texas or France, but not in England.

Additionally or alternatively, the externally available information mayalso be used to augment the generated text output options. For example,when a user input refers to an entity, externally available informationabout that entity can be acquired and, where appropriate, incorporatedinto generated text output options to enhance the depth and quality ofthe generated text. Acquisition of information from external sources maybe automatic as the user inputs information, or may be triggered by userinput. For example, the inclusion of a wildcard symbol such as a “?” mayprompt the writing assistant to acquire externally available informationfrom an external source, generate text based on the acquiredinformation, and insert the text in place of the wildcard symbol (or atleast provide text output options to the user for potential selectionand insertion at the site of the wildcard symbol).

The information available from external sources may also be used toensure that the text output options generated by the writing assistantalign with contextual aspects of preexisting text, user input, etc. Forexample, the external sources may be accessed to confirm the genderassociated with an individual identified in the preexisting text or userinput, to confirm facts about a referenced place name. to confirmchronology or dates, or (as previously mentioned) to verify the accuracyof facts or information. With the verification capability the writingassistant may generate text output options that may correct factualerrors included in the user input or that exist in preexisting text, forexample.

The external sources may be pre-selected by the user, be pre-set, orautomatically selected based on the user input or the attributesassociated with the user input. Relevant information in the externalsource can be identified automatically based on the attributesassociated with the user input. For example, if the user does not wantthe writing assistant to access the Internet, the user may block thatcapability. In this case, the writing assistant may call on informationthat is stored locally on a personal computer, smart phone, or otherclient device. In another example, the user may type in a name such as“Tony Johnson,” which the writing assistant will recognize as a name.Since the text includes a name, the writing assistant may access socialmedia accounts and available search engines to retrieve information thatmay be relevant to Tony Johnson, especially in the context of a documentbeing drafted. The writing assistant may, for example, find a “TonyJohnson” located in Paris, France (and may also use additionalinformation determined from the input or written text) to determine thatthis is the Tony Johnson being referred to by the user input orpreexisting text.

In some embodiments, the writing assistant may receive user inputincluding one or more words and, in response, retrieve information froman external source based on attributes associated with the user input.The attributes associated with the user input can be, for example, aname of a person, a place name, or an entity name. This list ofattributes is not meant to be limiting and could include any relevantattribute associated with the user input. The user input may alsoinclude a wildcard symbol. Common wildcard symbols include, but are notlimited to an asterisk (*), a question mark (?), etc.

The external source may be a local source or one that is housed on aremote network, another server, or another remote location. The externalsource could be, for example, a database containing geographicalinformation, entity information, organizational information, demographicinformation, physical property information, ontological information, orevent chronology information. The external source may also be a webpageor an electronic document accessible via the Internet.

The writing assistant may also receive user input including a collectionof two or more words that together convey certain ideas or facts. Asdiscussed above, the writing assistant may retrieve information from anexternal source based on the facts included in or implicated by thecollection of words. The facts associated with the user input caninclude, for example, a name of a person, a place name, or an entityname (e.g., “Paris” or “Tony Johnson”). This list of facts is not meantto be limiting and could include any relevant facts associated with theuser input. The user can include a wildcard symbol, such as ? or *, totrigger the system to collect information about the user input orrelative to a certain portion of the user input preceding or followingthe symbol. For example, a user may type “Tony Johnson?” or “*TonyJohnson” to prompt the writing assistant to search for information aboutTony Johnson. The writing assistant may, for example, search socialmedia for entries corresponding to Tony Johnson and, once the systemfinds a relevant profile, pull information from the profile about TonyJohnson, such as his city of residence, the high school he attended,recent likes. etc. The writing assistant can use the information fromTony's profile to augment suggested text output options.

In another example, a user may call the writing assistant and write“Bono's age is ?”, using the symbol ‘?’ to specify where a piece ofinformation should be retrieved and inserted in the sentence. Inresponse, the writing assistant may generate sentences such as “Bono is60 years old.”

In addition to freeform input, such as a series of words, the writingassistant can receive input from a user via one or more structured inputtemplates. Such structured input templates may facilitate entry ofinformation important to certain types of communications. A user maymanually select one or more templates to aid in information entry, orthe templates may be automatically triggered based on analysis of wordsentered by the user. For example, the user may choose, or the assistantmay detect and suggest, specific communicative intentions, such as“propose meeting” or “introduce someone.” This may initiate a dedicatedinteraction where the writing assistant is shown on a display and a usercan input the information or messages she wishes to convey in astructured or semi-structured manner.

FIGS. 6 a-6 o illustrate the template functionality that may beincorporated in or associated with the disclosed writing assistant. Asdescribed above, the user input may include words, phrases, sentences,etc. Within the user input, for example, the writing assistant mayrecognize certain words or phrases, for example, “meeting,”“information,” “request,” “buy,” “purchase,” or “task” associated withan available/predetermined input template. In response to a detection ofsuch keywords, the writing assistant may initiate one or more structuredinput templates to be shown on the display based on the detected word orphrase associated with a predetermined template. For example, as shownin FIG. 6 a , a user may open an email editor 605 and enter the name ofthe email recipient (i.e., the requestee 612 from whom the user isrequesting information). In this case, the user is composing an email to“Ernesto.” As shown in FIG. 6 b (and as described above), the user mayprompt the writing assistant for a user input field 61S. As shown inFIG. 6 c , the user may enter input 620 (“Please send me the”) intofield 615. The writing assistant may recognize a type 625 associatedwith the input 620 (in this case a request for information). Forexample, the writing assistant may recognize that the phrase “Pleasesend me the” most likely indicates that the user is sending the email torequest information from the requestee 612. In response, the writingassistant may suggest a type 625 of email to compose and mayautomatically display one or more predetermined templates determined torelate to the type of document being drafted or may display anindication, such as a detected type 625, that the user may select inorder to access available, relevant templates. In some cases, togetherwith an indication of a detected type 625 of document, the writingassistant may generate text output options 630 a and 630 b. It should beappreciated that the writing assistant can simultaneously provide theindication of a detected document type 625 together with the suggestedwell-written, context-fitting text output options 630 a and 630 b.

As noted, the user can select the suggested type 625, prompting thewriting assistant to display a predetermined template 680 associatedwith an information request, as shown in FIG. 6 d . The writingassistant may auto-populate some of the information in predeterminedtemplate 680. For example, based on the email address and greetingalready entered in the email, the writing assistant can determine that“Ernesto” (i.e., the requestee 612) will be the sender of the requestedinformation. And, the writing assistant may also automatically determinethat the user (“me”) is to be the recipient (i.e., the requestor 639) ofthe information and, in response, may auto-populate the Receiver field.The input 620 may also be inserted into the predetermined template. Thepredetermined template, in anticipation that the user will input theinformation that he is requesting, also may include an informationrequest filed 637 where the user can input the information that hewishes to receive from Ernesto.

As shown in FIG. 6 e , the user can input the information (e.g.,information input 643) into the information request field 637. Theinformation can be inputted in a variety of different ways. For example,as shown in FIG. 6 e , the user may enter “−avg weekly conversations &amounts” and “−team metrics—calls/hour” on separate lines. The writingassistant may analyze the information to determine the requestedinformation, despite the incongruent formatting and incompletesentences.

As shown in FIG. 6 e , additional, available input categories 640 a-640d may be displayed on the predetermined template 680. In this example,the additional input categories include purpose 640 a, deadline 640 b,urgency 640 c, and other requirements 640 d. However, it should beappreciated that these additional input categories may vary based on thetype of request, etc. The examples shown here are not meant to belimiting and only display a subset of possibilities.

As shown in FIG. 6 f , the user may select the input category purpose640 a. In response, as shown in FIG. 6 g , the writing assistant may adda purpose input field 643 to the predetermined template 680 along with apurpose suggestion 645. The purpose suggestion may be based on the textof the email or some other information. For example, the writingassistant could present a purpose suggestion of “present it in ourmeeting” based off a future meeting invitation with the subject “WeeklyTeam Meeting” where the user and Ernesto are both attendees, among otherrelevant information—external and internal—as discussed previously. Asshown in FIG. 6 h , the user can enter his own purpose input 647(“Quarterly report”).

As shown in FIG. 6 i , the user can select another input category, otherrequirements 640 d. As shown in FIG. 6 j , once the selection is made,another requirement input field 650 may be added to or displayedrelative to the predetermined template 680 (e.g., unhidden). And, likethe purpose input category, the writing assistant may display anotherrequirements suggestion 653 based on a similar methodology. As shown inFIG. 6 k , the user can add the other requirements input 655 (“don'tforget rick's team”) to the other requirements input field 655.

As shown in FIG. 6 l , the user can select another input category,deadline 640 b, prompting the writing assistant to add the deadlineinput field 657 to the predetermined template 680. And, like the purposeinput category, the writing assistant may display a deadline suggestion660 based on a similar methodology. As shown in FIG. 6 m , the user canadd the deadline input 663 (“tomorrow”) to the deadline input field 657.

As shown in FIG. 6 n , the writing assistant can use any or all of theinformation entered into the predetermined template 680 to create awell-written email that incorporates information entered into thetemplate to automatically generate a text output option 665 (e.g., “textoutput option 1”). Like the text output options described elsewhere inthis disclosure, the writing assistant may rely upon complete orincomplete sentences to create well-written text output options, whichmay be in the form of complete sentences. In this case, the text outputoption may include a greeting (“Ernesto,”) and a closing (“Thanks.”).

The user can modify or cause the writing assistant to refine text outputoption 665 in various ways. In some cases, the user may change a valueassociated with style parameter 667. For example, style parameter 667may correspond to a level of formality, but it can also include any ofthe previously discussed style parameters. In FIG. 6 n , the styleparameter 667 is set to “1.” As shown in FIG. 6 o , the style parameter667 can be changed to “2,” which may increase a level of formality of arefined text output option 670 (text output option 2) relative to textoutput option 665 (text output option 1). For example, the refined textoutput option may list the requested information numerically, mayinclude transitional phrases (e.g., from (“Also don't forget . . . ”) to(“Please make sure that . . . ”) and from (“i need it . . . ”) to (“Iwould appreciate it . . . ”), and may refine the closing (e.g., from“Thanks” to “Thanks for your help”).

The writing assistant can also display additional structured inputtemplates. For example, in some cases the writing assistant may displaya secondary structured input template based on secondary user inputreceived through the primary structured input template. And, through thesecondary structured input template, the user may input tertiaryinformation that conveys information with respect to a predeterminedsubject associated with the secondary structured input template. Suchtemplate generation may continue in a hierarchical or nested way suchthat additional templates may be displayed or made available to a userin response to any inputs included in a higher-level template. In suchembodiments, the writing assistant may automatically construct completesentence options that reference a predetermined subject and includeinformation conveyed by secondary user input. The complete sentenceoptions may also be automatically constructed to reference apredetermined subject of the secondary input template and to includeinformation conveyed by tertiary input. The complete sentence optionsmay differ from one another in at least one respect. The user can alsoenter a user-specified length for the complete sentence options.

The writing assistant may also be configured to automatically identifyinformation that may be missing from input that a user may provide tothe system, whether via a structured template or any other inputarrangement described herein. For example, the writing assistant mayreceive user input through a workspace. The user input can be acollection of words that convey at least one idea. Based on analysis ofthe user input, the writing assistant may detect the absence ofinformation that is not conveyed by the input but that may be relevantor important to the text or document being drafted. In such cases, thewriting assistant may prompt the user, through the writing assistantworkspace for example, to enter additional user input (e.g., secondaryuser input) associated with the missing information. For example, themissing information may include details like a time of a meeting, a timeof an event, a name of a person, a name of a place, a date associatedwith an event, a transaction amount, among many other possibilities.Through a structured input template or any other suitable interfaceelement, the writing assistant workspace may receive the secondary userinput that may include details associated with the missing information.The writing assistant may then construct complete sentence options orany other type of text output options that convey details includedwithin the secondary user input. All of the features described in thepreceding paragraphs with respect to the input methods, secondaryinputs, etc. can apply to this automatic identification of informationin any combination.

The writing assistant has the ability to iteratively interact with auser in order to refine or navigate through proposed text output optionsgenerated and displayed by the writing assistant. As shown in FIGS. 7a-7 f and as described above, the writing assistant can receive userinput and, in response, generate text output options. The writingassistant can display the text output options to the user who can selectone of the text output options for insertion into the document (e.g., inworkspace 710).

As For example, as shown in FIG. 7 a , a user can type text 712 intoworkspace 710 within email editor 705. As shown in FIG. 7 b , a user mayalso prompt the writing assistant to display a user input field 715where the user can enter input 720. Similar to other embodimentsdisclosed herein, the writing assistant may generate well-written,context-fitting text output options 725 a-725 c. As shown in FIG. 7 c ,the user can further interact with the writing assistant to refine anyof the generated text output options (e.g., by selecting virtual button730 corresponding to text output option 725 b). As shown in FIG. 7 d ,the writing assistant may use the selected text output 725 b to generateone or more refined text output options. For example, as shown in FIG. 7d , the writing assistant can display the selected text output option725 b (“The next action item is for us to elaborate our thoughts, andafterward discuss them with you.”) along with one or more refined textoutput options 735 a-735 c generated based, at least in part, on theselected text output 725 b. In other words, In this example, if for somereason the user was not satisfied with any of text output 725 a-725 b,the user may select any of the initially generated text output options(e.g., text output option 725 b) as the initially generated text outputoption closest to what the user envisioned for insertion into thedocument. In response, the writing assistant may generate one or morerefined text output options (e.g., text output options 735 a-c) based onthe user's selection from among the initially generated text outputoptions. This process may continue until the user finds suitable one ofthe generated, refined text output options.

In this example, the writing assistant may generate refined text outputoptions 735 a-735 c that seek to convey the same or similar meaning asthe selected text output 725 b, but have several differences relative totext output option 725 b. For example, the refined text output optionsmay include different introductory language (e.g., from “The next actionitem is . . . ” to “i think the next step is . . . ” or “I propose as anext step . . . ”), may include one or more synonyms (e.g., from “toelaborate . . . ” to “to further articulate . . . ” or “to refine . . .”), etc. As noted, this process may be iterative, and a user maycontinue request for refined text output options until he is satisfiedwith one of the options. For example, the user may select button 730 toprompt the writing assistant to generate further refined text outputoptions and so on.

As shown in FIG. 7 e , the user can select one of the refined textoutput options, such as text output option 735 a. As shown in FIG. 7 f ,the writing assistant can automatically insert the selected refined textoutput option 735 a into workspace 710. to create at least a portion ofthe email document.

The disclosed writing assistant may also assist a user in synthesizingmultiple text elements or text passages, whether available in one ormore preexisting documents or generated, in part, based on user input.In one example of such synthesis of text, and as described above, thedisclosed writing assistant may offer text output options for insertionat a selected location within a text. Such text options may serve tobridge or link text that may appear prior to and after the selectedinsertion point. This feature may be triggered manually, for example, bya user indicating a text insertion location in a document. The textinsertion location may be between two sentences, within a sentence,within a phrase, or between two paragraphs in the document. Thegenerated text output options may be generated based solely onpreexisting text appearing before and/or after the insertion location ormay also be based upon textual input provided by the user.

The text output options generated by the writing assistant forincorporating into a document at a selected insertion location may linktogether one or more aspects of a first text element that precedes thetext insertion location with one or more aspects of a second textelement that follows the text insertion location. For example, a textoutput option may be generated in such a way that it fits into existingtext in a coherent and natural way. The text output options can agreewith a context associated with the first and/or second text elements andmay, in some cases, be generated, in part, upon input provided by auser. For example, the generated text output options can include words,ideas, meanings, and topics conveyed by the user input, but may alsoagree with contextual elements associated with text preceding orfollowing a designated insertion location in order to effectively bridgeor link text surrounding the insertion location. The bridging text mayinclude a complete sentence or, in some cases, may include sentenceportions. For example, in some cases, the bridging text may include textto append to a preceding sentence, punctuation to end the augmentedpreceding sentence, one or more liking sentences, and/or text to appendto a beginning of a sentence following the insertion point.

Contextual agreement between the generated text output options andsurrounding text may have various meanings. In some cases, an agreementbetween two or more text elements may refer to grammatical agreement(e.g., the insertion of the generated text output option (the bridgingor linking text) does not result in a grammar error relative to thepreexisting text). In other cases, agreement between text elements maybe achieved by the generated text output options being generated toinclude in the same or similar style as the text around it (e.g.,preexisting text in a document workspace). Another contextual agreementmay exist where a generated text output option connects coherently tothe text around it once inserted into a document workspace. This form ofagreement may include, but is not limited to, the generated text beingrelated to the same general subject as the context and/or events orfacts referenced in a generated text output options being consistentwith events or facts referenced by preexisting text in a documentworkspace, for example. The consistency may be relative to arelationship (e.g., temporal, causal, teleological, explanatory, etc.)existing between generated text output options and preexisting text oruser input. Contextual agreement may also exist where facts implied bygenerated text output options are consistent with facts implied by thepreexisting text; where temporal and causal relations between facts orevents referenced in generated text output options and in thepreexisting text are not implausible in light of real-world constraints(e.g., a person can't perform an action after he dies. an event cannotstart after it ends, a person cannot be located in two differentlocations at the same time, etc.). As previously noted, a possible testof contextual agreement between preexisting text and text output optionsgenerated by the writing assistant may include whether more than seventypercent of human evaluators are not able to discern that a generatedtext output option, once inserted into the preexisting text, wasgenerated by a machine rather than by a human. In addition tocontrolling text style using style control parameters, the disclosedembodiments of the writing assistant may also be configured to apply adefault style that is predetermined or learned based on usage. Forexample, the writing assistant may learn the personal style of the useror the style of a particular organization, in different contexts (e.g.,based on internal business documents, external business email, personalemail, etc.). In this way, the writing assistant may generate suggestedtext output options to serve as linking or bridging text in a style thatresembles the personal or organizational style in the specific contextof the document.

In some cases, the writing assistant may automatically insertbridging/linking text into a document at the insertion location. In somecases, however, the writing assistant may generate and display multipletext output options, and the user may select a text output option, fromamong the displayed text output options, to be inserted into thedocument at the text insertion location. In response, the writingassistant may insert the user-selected text output option at theinsertion location.

Additionally or alternatively, the writing assistant may be configuredto synthesize text for a document based on other types of triggeringevents. For example, in some cases, the writing assistant mayautomatically generate bridging or linking text for insertion into adocument (or multiple linking or bridging text output options) based ondetected movement of one or more text elements from one location of adocument to another location. For example, in some cases, a user mayselect a portion of already drafted text to be moved from a firstlocation in the document to a second location in the document. The usermay drag and drop the selected text to the new location by highlightingthe text and dragging the text (using a pointer tool, for example) to anew location in the document. Alternatively, the user may use a cut andpaste function to cut text from one location in the document and pastethat text at a new location in the document. The user may also use acopy and paste function to copy text from an external source and pastethat text at a new location in the document. In such cases, pasting ofthe text in a new location may trigger operation of the writingassistant to automatically generate bridging or linking text relative tothe moved text and/or text surrounding the moved text. For example, oneor more modifications (word additions, word re-ordering, word omissions,new text, etc.) may be suggested relative to the moved text, and/orrelative to text preceding the moved text, relative to text followingthe moved text. In some cases the suggested bridging or linking text maynot involve changes to any of the preceding, following, or moved text,but instead may constitute new text passages to be inserted into thedocument before or after the moved text.

In some cases, the writing assistant may automatically assist the userwith a text move. For example, the writing assistant may include abuilt-in selection and move function that may be activated by, forexample, highlighting and right-clicking on the text. In response toreceipt of such input, the writing assistant may automatically identifya new location in the document for the selected text and may offer theuser an option for moving the highlighted text to the suggested newlocation. After the move, or in conjunction with the move, the writingassistant may generate one or more bridging text options associated withthe text move in the manner described above.

Thus, in response to any movement of text within a document, the writingassistant may automatically generate bridging or linking text outputoptions recommended to accompany the text movement. For example, whentext is transplanted from one document location to another, thetransplanted text often may not flow well with text in the vicinity ofthe new location. For example, the moved text may not agreegrammatically or contextually with surrounding text. To connect themoved text in a natural way, the writing assistant (in response todetected text movement within or between documents) may generate andoffer one or more text output options for insertion before or after themoved text. In some cases, the one or more suggested text output optionsmay include one or more modifications to the moved text to promoteagreement between the moved text and surrounding text at the newlocation.

The writing assistant may also be configured to synthesize text, whetherfound in existing text or included in user input, into more complex textpassages. For example, in some cases, a user may provide multiplesentences and/or sentence fragments as user input. The writing assistantmay organize and/or rearrange the input sentences or sentence fragmentsinto a logical order and may generate sentences, partial sentences, orparagraphs that convey ideas or information included in the inputsentences/sentence fragments, and may arrange the generated textaccording to the determined logical order. The text output generated bythe writing assistant may form a stand-alone text block that serve asthe first text associated with a document or that may be inserted intoexisting text in a document (either as a monolithic block or at leastpartially interleaved with text existing in the document). Wherefragments are received as input, the writing assistant may generatesentences based on the fragments and order the generated sentences toconvey information associated with the input fragments in a logicalorder. In any of the examples, sentences generated by the writingassistant based on input fragments may flow together in a coherent way.

In some embodiments, the writing assistant can take several pieces oftext, e.g., written by a user, or retrieved from other sources, andautomatically synthesize them into one coherent, fluent, and grammaticalpiece of text with a consistent style. For example, in an electronicworkspace associated with a document, the writing assistant may identifya first text passage, including a first plurality of words, and a secondtext passage, including a second plurality of words. The first or secondtext passage can be entered into the electronic workspace using a pastefunction initiated by the user, by the user typing on a keyboard ordictating using a voice recognition application, or by an electroniccopy function applied to a source of text residing outside of theelectronic workspace. In order to synthesize text from at least thefirst and second text passages, the writing assistant may change theorder of content in the text passages, merge sentences, split sentences,add connections between sentences or other portions of text, modifystyle elements, etc. Additionally or alternatively, the writingassistant may analyze the first and second text passages to determineinformation conveyed by the first passage and information conveyed bythe second passage and may use this information to automaticallygenerate a third text passage that conveys the information conveyed bythe first and second passages. The third text passage may includetextual revisions relative to the first and second passages. Forexample, the third passage may exclude a exclude words from the first orsecond passages and/or may include words not included in either of thefirst or second passages. Words from the first and second passages maybe, for example, reordered, merged, or substituted for new words in thethird passage. The third passage may include new text bridging words.The third passage may change style elements that were included in thefirst and second passages. In some cases, the writing assistant mayautomatically insert the synthesized third passage into a document ormay present the third passage to a user for approval and or refinement(e.g., using any of the interactive techniques described above).

In addition to a mode in which the writing assistant provides sentenceoptions as a user provides input, the writing assistant can also be usedto parse an existing document and offer text replacement options for oneor more sub-sentence elements or one or more complete sentences in thedocument. For example, users can select any span of text in theirdocument and call the writing assistant, which will automaticallygenerate, in real-time, several variations of well-written texts thatare paraphrases of the selected text. The user can choose any of theoptions and insert them to replace the selected text in the textbox orword processor. It should be noted that any or all of the featuresdescribed elsewhere relative to functionality of the writing assistantmay be used in the document parsing embodiments. For example, thewriting assistant may generate text output options as potentialreplacements for text elements automatically identified during theparsing operation. The user can use any of the described controls tochange various style parameter values associated with one or more of thegenerated options. The user can also select a particular text outputoption for insertion into the document in place of all or part of theidentified text. Further, the user can select a generated text outputoption as a trigger for causing the writing assistant to generate one ormore refined text output options based on the selected text outputoption (an interactive process that can continue until the user issatisfied with one of the generated text output options). Additionally,the user can enter additional input (e.g., one or more words) to helpguide the writing assistant in generating text output options (orrefined text output options) for potential substitution for textidentified during the automatic parse operation.

In some cases, automatically, or after receiving input from a user, thewriting assistant can analyze the text of a document. The analysis mayproceed in several ways, including sentence by sentence, among otheroptions. The parsing may be performed as part of a globalsearch-and-suggest operation.

Users can choose to view suggestions for sentences in their documentthat should be rephrased. Suggestions may be presented where theassistant can generate a paraphrase of any sentence in the documentwhich scores better in an automatic evaluation of metrics such asquality. clarity, grammatical correctness, etc.

The contextual paraphrasing feature of the writing assistant may helpusers refine their text by replacing words and phrases withsubstitutable alternatives—words or phrases that could substitute forgiven words or phrases such that the text remains fluent and its meaningis preserved (e.g., substitutable). The technology behind the featuremay close major gaps that exist when using lexical knowledge bases suchas thesauruses as sources for substitutable alternatives for words orphrases in text. For example, not all synonyms of a given word or phraseare substitutable in a given context, and not all words or phrases thatcan substitute for original words or phrases in a given context aresynonyms. In particular, synonym knowledge bases such as thesauruses arelimited in covering relations of semantic similarity between phrases.The contextual paraphrasing feature of the disclosed writing assistantmay provide both synonyms and non-synonyms that are substitutable in thegiven context.

For example, the writing assistant may suggest. autonomously or uponinvocation by the user, possible replacements of words or phrases in thetext with alternative words or phrases that are substitutable in theparticular context (such that after the substitution the text remainsfluent and its meaning is substantially preserved). The assistant mayalso recommend such replacements if they are determined to make the textmore fluent.

Replacements may include contextualized dictionary synonyms: words orphrases which are synonymous with the original word or phrase accordingto a lexical database, and are also found to be substitutable with theoriginal word or phrase in its particular context. For example, in ‘Iforgot all of the material I learned yesterday’, the assistant maysuggest replacing the word ‘material’ with the synonym ‘information’ (Iforgot all of the information I learned yesterday), because the twosynonyms are substitutable in this particular context. However, theassistant will not suggest the words ‘matter’ or ‘substance’ assubstitutions. because while they are synonyms of ‘material’, they arenot substitutable in this particular context. In ‘our brains preferinstant to long-term rewards’, the assistant may suggest replacing theword ‘rewards’ with the synonym ‘payoffs’ (our brains prefer instant tolong-term payoffs), but it will not suggest other synonyms such as‘bonuses’ or ‘prizes’ because they are not substitutable in theparticular context.

Replacements may also include contextualized possible substitutions thatare not lexical synonyms: words or phrases which are not lexicalsynonyms* of the original word or phrase, but are found to besubstitutable with the original word or phrase in a given corpusgenerally and in its particular context. For example, in ‘I enjoy doingYoga’, the assistant may suggest replacing the word ‘doing’ with theword ‘practicing’ (l enjoy practicing Yoga) (even though the words‘doing’ and ‘practicing’ are not recognized as synonyms). In ‘The pilotwas driving the airplane’, the assistant may suggest replacing the word‘driving’ with the word ‘flying’ (“The pilot was flying the airplane”)even though the words ‘driving’ and ‘flying’ are not synonyms. In ‘thankyou fir the good demo’, the assistant may suggest replacing the word‘good’ with the phrase ‘super useful’ (“thank you for the super usefuldemo”), even though they are not synonyms. For the purpose of thisdescription, words or phrases are not lexical synonyms of each other ifthat relation is not listed in common thesauruses. For example, twowords or phrases may be deemed non-synonymous if they are not related assynonyms in the following leading English thesauruses: Oxford Dictionaryand Thesaurus, Oxford Thesaurus of English, Longman Thesaurus ofAmerican English, Thesaurus of English Idioms. Collins EnglishDictionary and Thesaurus Set, Webster's American English Thesaurus,Roget's Thesaurus of English Words and Phrases, www.thesaurus.com,www.macmillanthesaurus.com, and/or The Merriam-Webster Thesaurus.

Word or phrase substitution suggestions made by the disclosed writingassistant could be of different lengths from the original words orphrases. For example, the assistant may suggest replacing a word with aphrase, a phrase with a phrase of a different length, or a phrase with aword. For example, the assistant may suggest replacing ‘All in all, Ithink we are ready’ with ‘Taking everything into account, I think we areready’. The assistant may suggest replacing ‘rights ought to beprotected against infringement’ with ‘rights should be protected againstinfringement’. The assistant may suggest replacing ‘If you work hard youcan change things’ with ‘If you work hard you can make a difference’.Suggestions may sometimes include substitutions that are not synonyms(according to lexical knowledge bases) of the original text but canreplace the original text in the particular context while substantiallypreserving the meaning of the sentence as a whole. The assistant mayprovide completely different substitution suggestions for the same wordor phrase in different contexts or contextual situations.

The technical method may include two components. First, it may include acomponent that may curate a static list of possible replacements forwords or phrases. Second, in a given call to provide paraphrasingsuggestions, the writing assistant may include a component that presentsonly the words or phrases from the static list determined to constituteappropriate substitutes for the original word or phrase in the givencontext. Words or phrases deemed to not constitute appropriatesubstitutes in view of the context in which the original word or phraseappears may be omitted from the output results of the paraphrasing tool.

The curation of a static list of substitutable candidates may includecollecting lexical synonyms for each word or phrase from a thesaurus orcollecting possible corpus-dependent replacements for words or phrases,in the following ways: (1) extracting a plurality of sentences where theword appears in the corpus (e.g., each sentence may provide an example“context” for the word or phrase); (2) for these contexts, a MaskedLanguage Model (e.g., BERT) may be used to mask the word and attempt topredict it; (3) keep X (in the 100 order of magnitude) contexts wherethe MLM successfully predicts the masked word or phrase according to athreshold: (4) for these disambiguating contexts, we may look at theother words or phrases which are predicted by the MLM; (5) we may ignoreknown antonyms of the given word or phrase, as they appear a lot in thesame context (“I adore old films” or “I can't stand old films”) but arenot appropriate replacements of each other. These 100 contexts can thenbe seen as “disambiguating contexts,” ones from which it is possible todeduce the correct word. We do this to avoid contexts of the form “1made a cake” for the word “cake”—a context where there are many wordsthat could replace “cake”, a negligible amount of which are actualreplacement options for “cake”. However, a context like “I baked achocolate cake for the party” is one where “cake” would be a reasonableprediction, and other reasonable predictions are indeed similar (“pie,”,“muffin,” etc.). The words or phrases which are predicted together withthe original word or phrase enough times are considered to be thecorpus-dependent contextualized replacement candidates. In summary, thecorpus-dependent replacement options may include words or phrases whichoften appear in similar disambiguating contexts as the original word orphrase, thus sharing some sense with the word.

Upon a given call to suggest replacements for a word or phrase, thesystem contextualizes the replacement suggestions (i.e., the system maypresent as text options only the suggestions from the static list thatare determined to be substitutable with the original word or phrase inthe particular context associated with the original word or phrase orthe text in which the original word or phrase appears). To do this, wemay use the paragraph written by the user as context which we feed intoour MLM, masking the word or phrase that the user wishes to replace. OurMLM gives us a list of predictions for the masked word or phrase, whichwe then intersect with the static list of replacement suggestions. Theintersection of these two lists are meaningful replacements for thegiven word that are also substitutable with the original word or phrasein the particular context, and these are the suggestions that areprovided to the user.

FIGS. 8 a-8 d illustrate another example of functionality that may beincluded in the disclosed writing assistant. As shown in FIG. 8 a , thewriting assistant can identify a first drafted text element 820 inpreexisting body text 815 in workspace 810. Drafted element 820 mayinclude portions of two or more sentences or a group of words within asentence. The writing assistant may automatically highlight the firstdrafted text element on the display. or a user may manually highlightthe element to be edited by the writing assistant. As shown in FIG. 8 b, the writing assistant may generate text output options 835 a and 835 bthat re-write the first drafted text element 820, fit the context of thebody text 815, can be placed in the same location as the first draftedtext element 820, and convey a meaning associated with the first draftedtext element. As shown in FIG. 8 c , the user can select one of the textoutput options (e.g., option 835 b). As shown in FIG. 8 d , the writingassistant may automatically replace first drafted text element 802 withthe selected text output option 835 b.

The writing assistant can repeat this procedure for multiple draftedtext elements, as shown in FIG. 8 a (e.g., for an automatically ormanually identified second drafted text element 825 and third draftedtext element 830). In FIG. 8 a , the second drafted text element 825 andthird drafted text element 830 occur after the first drafted textelement 820. However, because of the iterative nature of thisembodiment, the second or third drafted text elements could haveoccurred before the first drafted text element. This procedure cancontinue with third, fourth, fifth, etc., text elements and is notlimited to the identified text elements described in this example.Additionally, the described process may be iterative, so that once thewriting assistant parses through the document once, even if the usermakes suggested changes, the writing assistant may detect additionaldrafted text elements to be revised, which may be located anywherewithin the modified document.

For example, a user may highlight one or more sub-sentence elements orsentences in an existing text, and in response, the writing assistantmay generate one or more alternative text options for possiblesubstitution for any of the highlighted text. The text output optionsmay be synonymous or not synonymous with the first drafted text element,or a portion thereof. They can also be generated as a replacement forthe first drafted text element, or a portion thereof, or to agree withat least one contextual element associated with text in the documentother than the first drafted text element. The text output options caninclude complete sentences and may include more or fewer words that thedrafted text element. In some cases, the generated text output optionsmay include no words from the first drafted text element. The textoutput options may also include one or more changes relative to thefirst drafted text element, a change in verb tense, an addition of atleast one clause, or a substitution of one or more synonyms relative tothe first drafted text element. The changes relative to the firstdrafted text element can include, for example, a style modification, agrammar modification, or a modification of words included in the firstdrafted text element.

As in previously described examples, the writing assistant can receive auser selection of a text output option and automatically insert theselected text output option into the document text in place of at leasta portion of the first drafted text element. If there are two or moretext output options, then the writing assistant can use the selectedtext output option to further refine and update the text output options(e.g., based on user selection of a GUI control associated with a textoutput option refinement process).

Various controls may be used to initiate and/or control the presentlydisclosed writing assistant system. For example, as discussed in thesections above, one or more GUIs associated with the writing assistantmay include virtual buttons (e.g., icons, etc.), menus (e.g., drop downmenus), among other virtual control elements that a user can interactwith to control various aspects of the writing assistant. For example, avirtual control button may be included to initiate operation of thewriting assistant. As shown in FIG. 4D, fields and buttons may beincluded in a GUI to select controllable style parameters and set valuesfor the control parameters. Other buttons may control selection andinsertion of a generated text output option into a workspace. Variousother virtual buttons, fields, menus, etc. may be included foraccomplishing any other tasks associated with the writing assistant.

In some cases, other types of user interface elements may be used tocontrol one or more aspects of the writing assistant. Such interfaceelements may include, for example, a keyboard 902, as shown in FIG. 9A,a mouse or other pointing device, electronic pencil, etc. that mayinclude one or more controls adapted to enable a user to interact withthe writing assistant.

As shown in FIG. 9A, keyboard 902 may include a button 904 (“Assist”)that when pressed may initiate the writing assistant. For example,continuing with the example of FIG. 4 above, a user may wish to make acall to the writing assistant at any time while drafting a email orother type of electronic text-based document. Before or after enteringtext into a workspace 912, a user may initiate the writing assistantfunctionality by pressing button 904, which may result in a user inputfield 914 being shown on the GUI display, as shown in FIG. 9B. Userinput field 914 may include any or all of the functionality describedabove relative to other user input fields. For example, in response toone or more words being entered into user input field 914, the writingassistant may generate and display one or more text output optionsassociated with the one or more words entered into field 914.

Other controls may be included on keyboard 902. For example, a button906 (“Style”) may be used to cause the writing assistant to display oneor more GUI elements associated with selection of available styleparameters and associated style parameter values. For example, in somecases, after initiating operation of the writing assistant, a user pressbutton 906 to set values for various style parameters to be usedglobally by the writing assistant in generating text output options.Style button 906 may also be used to select style parameters to beapplied more locally. For example, a user may select/identify aparticular text output option generated by the writing assistant (e.g.,by highlighting the text output option or clicking on a virtual button,etc. associated with the text output option) and press Style button 906in order to select and/or change one or more values associated withavailable style parameters for the particular text output option.

In other cases, a user may highlight text in a document (with or withoutthe writing assistant being active) and press Style button 906 in orderto select/set available style parameter values for the highlighted text.For example, a user may highlight a word, phrase, sentence, etc., andthen press button 906. In response, the writing assistant mayautomatically be initiated, and a GUI may be displayed to enable theuser to set various style parameter values associated with thehighlighted text. In response to a selection/change in style parametervalues and/or in response to any suitable user input (e.g., pressing oneor more virtual buttons, pressing the Enter key, etc.), the writingassistant may generate one or more text output options generated basedon the selected style parameter values as potential substitutes for thehighlighted text.

In some examples, pressing button 906 may cause the writing assistant todisplay a GUI, as shown in FIG. 9C, for enabling a user to select ormodify one or more style parameter values. Once displayed, the user mayselect an available style parameter or enter a value for a particularstyle parameter using various control elements associated with the GUI.For example, a user may place a cursor within any of input boxes 918a-918 d in order to enter a specific value associated with each styleparameter or to activate a drop-down menu of available values, which canthen be selected. Alternatively, a user may use+/−buttons 916 a-916 d(or any other suitable control) to increase or decrease particular styleparameter values. While the GUI of FIG. 9C shows style parametersincluding Politeness, Formality, Emotion, and Conciseness, any otherstyle parameter value may be used by the described writing assistant.For example, in some cases a Length parameter for controlling a lengthof generated text output options may be grouped together with otherstyle parameters.

Additionally or alternatively, one or more other control elements may beused for controlling various features of the writing assistant. Forexample, as shown in FIG. 9A, a keyboard 902 may include directionalarrow keys 908 and a scroll wheel 910. Other input devices, such as amouse or electrical pencil may include similar features such as arotating wheel, up/down buttons, touch sensitive “buttons”, etc.Returning to the style parameter example, keys 908 and wheel 910 may beused to select/change style parameter values. For example, when a styleparameter control GUI, such as the GUI shown in FIG. 9 c , is madeavailable to a user, the user may select a particular style parameter toupdate by pressing the left or right directional keys 908 to cyclethrough the available style parameters. Once the desired style parameteris reached, the user may turn the scroll wheel 910 to change the valueof the style parameter (e.g., turning left to decrease the value andturning right to increase the value). After selecting a desired valuefor a style parameter, the user may press wheel 910 (or hit the Enterkey) to update the style parameter with the selected value.

Alternatively, in some cases, directional keys 908 may be omitted, andwheel 910 may be used to control a combination of features. In the styleparameter example, a user may turn wheel 910 left or right to cyclethrough the available style parameters shown in the GUI of FIG. 9C. Oncethe desired style parameter is reached, a press to wheel 910 may enablea value selection function for the style parameter. In such a case,turning wheel 910 to the left may decrease the value and turning wheel910 to the right may increase the value. After selecting a desired valuefor a style parameter, the user may press wheel 910 (or hit the Enterkey) to update the style parameter with the selected value.

Keys 908 and wheel 910 (and any other included control elements) may beused to interact with any features and functions associated with thedisclosed writing assistant. For example, keys 908 and/or wheel 910 maybe used to scroll through available menu items or GUI elements, selectvarious options or parameter values, etc. While the example keyboard 902shown in FIG. 9A includes controls 904, 906, 908, and 910 included in adedicated region of the keyboard, any suitable arrangement of thecontrols may be used. In some cases, buttons 904, 906, and 908 (andwheel 910) may be distributed over different areas of keyboard 902. Insome cases, the described functionality associated with buttons 904,906, and 908 (and wheel 910) may be associated with one or more otherbuttons of keyboard 902, such as an of the Function keys, directionalarrow keys, etc.

One aspect of the writing assistant may include the generation ofnatural language that may be controlled or influenced by multiple piecesof text that should be naturally and smoothly incorporated into arefined text passage or text output option. There may be varioustechniques for assembling a writing assistant application consistentwith the presently disclosed examples and embodiments. In some cases,the disclosed writing assistant may be assembled and/or configured usingmachine learning techniques and/or by incorporating one or more trainedmodels. In order to provide the described functionality, the disclosedwriting assistant and model(s) on which the writing assistant is basedmay be trained, for example, to predict text within a document from alarge corpus, conditioned upon text appearing before and/or aftertextual elements. For example, in order to train the model(s), one ormore large text corpus documents (such as one or more of severalpublicly available corpus documents) may be segmented into sentences.Such sentences may be randomly selected and revealed to the model(s) toserve as context for predicting the text in the other sentences withinthe document (e.g., sentences that appear in close proximity to arandomly selected sentence). The model(s) may thus learn to generatewords conditioned on the multiple pieces of text provided by the userand to generate words, sentences, etc. that fit within contextestablished by text in a document.

As one example of training a model on which the disclosed writingassistant may be based (e.g., a training method for autoregressiveleft-to-right language generators) may include selective masking ofvarious portions of a corpus document. In some cases, such documentsused for training may include just a few sentences or paragraphs. Inother cases, however, such documents may be thousands or hundreds ofthousands of pages long and may offer many examples of word usages,context dependencies, etc. When constructing a training set using atraining document, portions of the document may be labeled to obtain twoparts (e.g., a prefix and a suffix). In some cases, such splits may beintroduced at the end of a sentence within the training document. Theprefix begins at the beginning of the training example and ends at thebeginning of the suffix, which ends at the end of the example. Thetraining example may then be re-ordered to place the suffix tokens(e.g., text portions) at the beginning of the sequence, followed by asequence-start token, the prefix tokens and a sequence-end token. Withthis technique, the model(s) may be trained to predict the tokens of theprefix while being exposed to the tokens of the suffix.

Another aspect of a method for training model(s) associated with thedisclosed writing assistant may include training techniques to control adesired length of the generated text, while ensuring that the generatedtext does not end abruptly, but rather concludes in a natural way. Oneway to do this is to train the model to predict text within a documentfrom a large corpus conditioned upon the length of ground-truth text inaddition to other signals, such as preceding text.

For the same autoregressive setting discussed above, this may beaccomplished by assigning each token with a positional embedding priorto re-ordering each training example, such that the suffix tokens encodetheir true position in the full text, and therefore indicate thegeneration length as well. Optionally, the positional embeddings can berandomly shifted by a small amount. To handle cases where the generationis not conditioned on the suffix, the generation length may be encodedin the positional embeddings of the start-sequence token. The model(s)may thus learn to generate tokens conditioned on the length and positionof text that should be generated.

Another aspect of training for the model(s) associated with thedisclosed writing assistant may be directed to enabling the model(s) todetermine a desired position of generated text within a predeterminedtext (e.g., such that the generated text is incorporated naturally andsmoothly within the preexisting text). Such capabilities may be providedby training a model to predict text within a document from a largecorpus conditioned upon the preceding text and additional informationregarding the position of the missing text. In addition to the methoddescribed in the previous section, after converting the tokens into acontinuous representation, a representation denoting the original indexof each token may be added. The model(s) may thus learn to generatewords conditioned on the length and position of text that should begenerated.

Another aspect of model training may be directed to the generation ofnatural language that conveys a desired meaning. The desired meaningcould be indicated by, among other things, the following: naturallanguage phrases or sentences that express the desired meaning or intentfor the meaning of the generated text; keywords that express the desiredmeaning or intent for the meaning of the generated text; any indicationof semantic objects and relations that should be included in thegenerated text. such as entities (e.g. people, locations, events, etc.),relations between events (e.g. temporal, spatial, cause-effect, etc.),relations between entities (e.g. organizational, family, etc.),relations between entities and events (e.g. winner-lottery,seller-purchase, etc.).

Below is a description of a method for training a language model tocapture relations between weak semantic signals and surface text. Themodel may be trained to predict masked spans of text in a large corpusconditioned upon the textual context and upon semantic signalsautomatically extracted from the masked text, which may simulate signals(in user input or extracted from the input) that indicate the desiredmeaning of the generated text at prediction time. The model may thuslearn to generate text that expresses the meaning indicated by the inputat prediction time. Semantic signals that could be extracted from themasked text may include, but are not limited to, surface semanticphenomenon, representations of semantic meaning, and/or heuristics fortransforming sentences into broken or simple forms, including but notlimited to, Machine Translation into Simple English, insertion ofgrammatical mistakes, etc. Surface semantic phenomena may include, butare not limited to, a bag of words (e.g., a set of meaning-carryingwords that are used in a particular sentence), synonyms, and paraphrasesof a particular sentence, that could be generated. among other methods,by back-translation. Representations of semantic meaning may include,but are not limited to, extraction of semantic frames and roles (e.g.,[frame: purchase; roles: {buyer: ‘john’; seller: ‘Tod’; object:‘car’}]); extraction of entities (e.g., persons, events, locations,etc.); extraction of sentiments (e.g. positive, negative); extraction ofdependency parsing, extraction of discourse relations between phrases(e.g., contrast, example, elaboration, etc.); word senses; wordembeddings; extraction of speech act illocution or intent (e.g. ‘proposemeeting’, ‘agree to suggestion’, etc.); and learned latent semanticrepresentation.

One level of semantic meaning that may be considered is the clauselevel. In use, it would be desirable for the model(s) of the writingassistant to generate text conveying the same or similar meaning as theuser input (or selected, preexisting text). In order to accomplish this,a semantic representation may need to capture the meaning of the userinput clause-by-clause and to capture the relation between the clauses(e.g., equality, entailment, description, etc.). In addition, semanticequality can be provided at a higher resolution. For example, it may berequired that the properties of the entities will be maintained betweenthe user input and the generated text, e.g. the gender or age of thesubject. In order to accomplish this, the semantic representation of theentities for the properties to be conserve may be queried.

In some examples, learning to condition on a semantic representation maybe accomplished in two steps: supervised and unsupervised. In thesupervised step, a dataset of annotated examples may be leveraged totrain a model (“Semantic Reader”) on a few Natural LanguageUnderstanding tasks which capture semantics (such as Semantic RoleLabeling, Semantic Proto-Roles, Coreference, Entity Linking, etc). Oncetrained, the Semantic Reader may be applied to a large corpus producingpredictions for the different semantic tasks. In the unsupervised step,another model (e.g., a “Semantic Generator”) may be trained to generatemasked text conditioned on the output of the Semantic Reader.

In addition, the Semantic Reader can be applied again to the output ofthe Semantic Generator for training examples in the unsupervised step,and the Semantic Generator may be trained to minimize reconstructionloss on the output of the Semantic Reader. Optionally, the SemanticReader weights may be updated as well.

Another training method for the disclosed writing assistant models mayinclude determining the desired meaning of generated text. Such adetermination may be accomplished by using sampling methods from thelanguage model guided by certain constraints and derived from thefollowing metrics (among others): diversity of vocabulary, diversity ofsyntactic structures, the semantic similarity to the input, style,coherence, and fluency. Text generation based on a language model mayrequire sampling from a provided probability distribution. The desiredoutput should be likely and must rank high in terms of the abovemetrics. Finding an desired solution may be intractable for anyreasonable generation length, so a sub-optimal algorithm may be employedthat can provide an approximation. An automatic evaluation of theabovementioned metrics may be used to guide the sampling from thelanguage model.

Another method for determining the desired meaning of generated text mayinvolve training the language model with reinforcement learning wherethe model's reward is derived from any of the abovementioned metrics,for example. While training a model to predict a masked word, the modeltrained up to this step may be used to generate text as well. Errorsfrom the text generation step may be propagated into the model trainedto predict a masked word.

Another method of for training the model(s) of the writing assistant todetermine or generate a desired meaning of generated text may includeenriching text generation by using external knowledge bases. Suchexternal knowledge bases may relate to (among other things):geographical KB—spatial relations; organizational KB such as CRM:demographic kB: ontologies; physical properties KB; Wikipedia;historical knowledge; and event graphs. Such external knowledge basesmay be used, for example, to ensure semantic coherence of the generatedtext. For example, an agent could be in Paris and France at the sametime but not in Paris and England. For this use, both in the languagemodel training phase and in the text generation phase, we can verifythat the generated text doesn't contradict the external knowledge (i.e.,for text generated we will extract facts and verify that they arealigned with information from the external knowledge base).Additionally, the external knowledge bases can be used to improve thequality of the generated text by augmenting it with information from anexternal knowledge base or appropriately replacing certain informationor object references. For example, when the generated text should referto an entity that exists in the external knowledge base, we can replacethe user's reference with an alternative reference to the same entity oradd information on that entity found in the knowledge base.

Another method for generating text with the desired meaning may includeusing a semantically infused language model for text generation. Forexample, a neural network-based language model may be trained to containcontextual relations between abstract semantic features in text, incontrast with prior systems, where models can only be trained to learncontextual relations between surface-level words. For example, thepresently disclosed writing assistant may include model(s) trained tolearn contextual relations between words and word senses and betweenwords and the properties of the abstract concepts invoked by the text.To achieve this, a model may be trained to predict the semantic featuresof masked tokens in text conditioned by their surrounding context. Usinga semantically infused language model to generate text may improve itssemantic coherence and plausibility. Such methods may allow us to endowthe language model with a semantic signal given unlabeled text only,which may result in an ability to harness information from massiveamounts of raw text.

The disclosed system and method may allow for endowment of a languagemodel with a semantic signal given unlabeled text only, thus enjoyingthe ability to harness information from massive amounts of raw text. Thedisclosed trained language models, infused with such semantic knowledgegained from pretraining, may achieve enhanced performance on naturallanguage tasks with merely a fraction of parameters compared with othersystems. Types of semantic signals that could be infused into languagemodels using the following described technology may include: using themethod described above to learn contextual relations betweensurface-level words and additional semantic features, including wordsenses: real-world properties of concepts invoked by the text (e.g.size, color, etc.); entity types (e.g., organization, person, animal,feeling, etc.); entity links (what different words refer to the sameentity described in the text); the sentiment (e.g. positive, negative,neutral); discourse relations between phrases (e.g. contrast, example,elaboration, etc.); and multiword expressions (the sense of multiplewords taken together). Word senses can include a system and method forthe generation of a semantically infused language model that capturescontextual relations between words and word senses and supersenses. Themodel may be trained to predict word senses of masked tokens in a corpusgiven the textual context. The ‘correct’ word senses may be derived froman ontology or a lexical knowledge base such as Wordnet.

An additional component of the system and method may include enforcingprediction coherency. Having extended the pretraining setting to amultitask one, where semantic information is predicted in parallel tosurface-level word information, we developed a global consistencyconstraint validation procedure. We effectively enforce the predictionsof the different semantic tasks to be consistent with one another. Forexample, an independently predicted pair of word and sense for a maskedposition should be plausible (e.g., the predicted word could have thatsense, a predicted part-of-speech label should be consistent with anindependently predicted parse tree structure, etc.). The process mayincrease the accuracy of semantic information prediction.

Additionally, the system and method may allow for infusing a languagemodel with semantic features through a model's loss function. Weformulate the loss function when training a masked language model suchthat the model is rewarded to some extent for predicting hypernyms andsynonyms of the masked words, and not merely for precisely predictingthe word. Specifically, our loss function is “forgiveing” in anexponentially decaying manner as a function of the distance of thepredicted words from the masked word in the WordNet graph. For example,it punishes predictions of WordNet synonyms, hypernyms, or hyponyms ofthe masked words much less than it punishes predictions of unrelatedwords.

Additionally, the system and method may allow for saving time and moneyby using micro BERT models, and then scaling up. We developed a gradualpretraining strategy where various hyperparameter ablations areperformed on significantly smaller and cheaper models, and only thenleading experiments are performed on common expensive models.

Automated (or semi-automated) text generation holds great promise forsociety, by helping people write better and more productively. In orderto unlock this potential, however, text generators need to evolve tobecome more controllable. Impressive as it is, text generated by priorsystems is far from perfect. In particular, the prior models' outputtends to diverge from the human-written input as the generationprogresses. Sooner or later, the prior generators go off-topic, losecoherence, or contradict the preceding text. Such behaviors are a majorproblem for a user trying to convey a message or express an idea.

There is no natural way for a user to restrict this tendency to divergein the outputs of prior language generation systems. This divergence,for example, is inherent to their left-to-right, extrapolating method ofoperation. Metaphorically speaking, the user can give these models astarting point and a vague sense of direction, but not a finaldestination, let alone a route to follow.

The disclosed writing assistant is designed to enable a user toeffectively control the “route” used by the writing assistant ingenerating its text output options. And as described in the sectionsabove, if a user does not feel that the system has reached the intended“final destination” by offering a text output option that conveys anintended meaning, information, etc., the user can provide additional ordifferent directions about the route until the writing assistantmetaphorically reaches the intended final destination. Such control isnot offered by prior language generation systems.

To provide this type of controllability, the disclosed writing assistantmay be based upon an interpolating language model. That is, given ahuman-written beginning (prefix) and human-written ending (suffix), thewriting assistant can generate synthetic text (body) that fits betweenthem with a desired length. Thus, the writing assistant may offer atleast two new “knobs” for tuning its output: the suffix, for keeping thegenerated text on topic, and the length, for controlling the amount oftext inserted between the prefix and the suffix.

In some cases, the writing assistant may be trained relative to publiclyavailable text.

For example, one or more models associated with the disclosed writingassistant may be trained on OpenWebText, a freely-available clone ofOpenAI's WebText dataset. In order to train the model to generate textconditioned on a prefix and a suffix, the order of the text may bemanipulated in different training examples.

What follows is a more technical description of an exemplaryimplementation of aspects of the writing assistant. For example, in somecases, the disclosed writing assistant may be based on a model with 24layers with 16 attention heads and 1024-dimensional hidden states, whichamounts to 345 million parameters. The same vocabulary and BPEtokenization scheme may be employed. One goal may include providing agenerative model of natural language allowing for sampling according tothe conditional distribution:P(x _(p+1) , . . . ,x _(n−s) |x ₁ , . . . ,x _(p) ;x _(n−s+1) , . . . ,x_(n))

where (x_(i))_(i=1) ^(n) is a sequence of tokens, (x_(i))_(l=1) ^(p) isthe prefix, (x_(i))_(i=n−s+1) ^(n) is the suffix and (x_(i))_(i=p+1)^(n−s) is the body. For comparison, certain prior systems sample fromP(x_(p+1), . . . , x_(n)|x₁, . . . , x_(p)), conditioned only on theprefix tokens, with some also sampling on additional metadata fields.

The disclosed writing assistant may adopt an autoregressive formulationof language modeling, decomposing the probability of a sequence(x_(i))_(i=1) ^(n) into a product of the conditional probabilities ofgenerating each token given the previous tokensP(x _(p+1) , . . . ,x _(n−s) |x ₁ , . . . ,x _(p) ;x _(n−s+1) , . . . ,x_(n))=Π_(l=p+1) ^(n−s) P(x _(i) |x ₁ , . . . ,x _(i−1) ;x _(n−s+1) , . .. ,x _(n))

To condition the output on the suffix, the input sequences can bearranged such that the first s tokens are the suffix, followed by theprefix, separated by <begin> and <end> tokens. In order for the model toproperly “stitch” the generated text to the suffix, the startingposition of the suffix may be indicated, thereby dictating the sequencelength. This can be done by assigning the suffix (prefix) tokens withpositional embeddings corresponding to their original positions at theend (beginning) of the sequence, rather than their position in therearranged sequence.

The model may be trained to minimize the cross-entropy loss whenpredicting the input sequence. In some cases, backpropagating the losson the suffix tokens, corresponding to the first s tokens in the inputsequence, may be avoided. The training sequences may be generated asfollows:

-   -   1. For each document in OpenWebText, we can sample [N/n_(max)]        sequences of consecutive sentences (Sentok may be used, in some        cases, for sentence segmentation), where N is the total document        length. The sampled sequence length n, including two special        tokens (<begin> and <end>), is uniformly distributed in        [n_(min), n_(max)]. We set the minimum and maximum sequence        lengths as n_(min)=32 and n_(max)=512 tokens respectively.    -   2. For each sequence, we can extract a suffix containing m        sentences from the end, such that m is uniformly distributed in        [1, min(M−1,m_(max))], where M is the total number of sentences        in the sequence. Thus, at least one sentence is reserved for the        prefix. We trained with at most m_(max)=3 sentences in the        suffix. To train the model to be able to predict given only a        prefix, we didn't extract a suffix for 10% of the sequences.    -   3. The final input sequence may be composed by concatenating the        extracted suffix tokens, a <begin> token, the prefix tokens and        finally an <end> token.    -   4. The first s+1 tokens, corresponding to the entire suffix and        the <begin> token, may be assigned positions n−s+δ to n+δ        (inclusive). The remaining tokens, corresponding to the prefix        and the <end> token, are assigned positions 1 to n−s−1        (inclusive). The random shift δ is introduced to soften the        length constraint, effectively allowing the model some leeway at        inference time. We sampled the position shift uniformly in        [0,0.1×n].

The model may be refined using Adafactor and certain hyperparameters.For example, a learning rate schedule may be used with a linear warmupover the first 10,000 steps to a maximum learning rate of 3×10⁻⁴followed by linear decay over the remaining steps. The model may betrained for 800,000 steps with a batch size of 512, corresponding toapproximately 20 epochs over OpenWebText. Training, in some examples,took roughly 3 days on a 128-core TPUv3 pod. At the end of training, theloss on both the training set and a held-out set continued to decrease,so further training may improve the model's performance.

As additional context for the disclosed writing assistant and itscapabilities, the ability to learn from large unlabeled corpora hasallowed neural language models to advance the frontier in naturallanguage understanding. However, existing self-supervision techniquesoperate at the word form level, which serves as a surrogate for theunderlying semantic content. The disclosed writing assistant is based ontechniques employing weak-supervision directly at the word sense level.In some cases, a model on which the writing assistant may be based maybe referred to as SenseBERT, which is a model pre-trained to predict notonly the masked words (as described above) but also their WordNetsupersenses. As a result, the disclosed writing assistant may be basedon a lexicalsemantic level language model, without the use of humanannotation. SenseBERT may achieve significantly improved lexicalunderstanding, as compared to prior systems.

One starting point for the disclosed model and its training may includethe use of self-supervision, which may allow the network to learn frommassive amounts of unannotated text. As noted above, oneself-supervision strategy may include masking some of the words in aninput sentence and then training the model to predict them given theircontext. Other strategies for self-supervised learning may include, forexample, unidirectional, permutational, or word insertion-based methods.

The disclosed writing assistant may be based on models that applyweak-supervision directly on the level of a word's meaning. By infusingword-sense information into a pre-training signal (e.g., a BERTpre-training signal), the model may be explicitly exposed to lexicalsemantics when learning from a large unannotated corpus. The resultantsense-informed model may be referred to as Sense-BERT. For example, amasked-word sense prediction task may be added as an auxiliary task inBERTs pretraining. Thereby, jointly with a standard word form levellanguage model, a semantic level language model may be trained thatpredicts the missing word's meaning. This method does not require senseannotated data. Self-supervised learning from unannotated text may befacilitated by using WordNet, an expert constructed inventory of wordsenses, as weak supervision.

The disclosed models and their training may focus on a coarse-grainedvariant of a word's sense, referred to as its WordNet supersense, inorder to mitigate an identified brittleness of fine-grained word-sensesystems, caused by arbitrary sense granularity, blurriness, and generalsubjectiveness. Word-Net lexicographers organize all word senses into 45supersense categories, 26 of which are for nouns, 15 for verbs, 3 foradjectives and 1 for adverbs. Disambiguating a word's supersense hasbeen studied as a fundamental lexical categorization task. In thedisclosed embodiments, the masked word's allowed supersenses list fromWordNet may be employed as a set of possible labels for the senseprediction task. The labeling of words with a single supersense (e.g.,‘sword’ has only the supersense noun.artifact) is straightforward. Thenetwork may be trained to predict this supersense given the maskedword's context. As for words with multiple supersenses (e.g., ‘bass’ canbe: noun, food; noun, animal; noun, artifact; noun, person; etc.), themodel may be trained to predict any of these senses, leading to a simpleyet effective soft-labeling scheme.

Compared to prior systems, the disclosed models on which the writingassistant may be based may significantly outperform those systems by alarge margin on a supersense variant of the SemEval Word SenseDisambiguation (WSD) data set standardized in Raganato et al. (2017).Notably, SenseBERT receives competitive results on this task withoutfine tuning: i.e., when training a linear classifier over the pretrainedembeddings, which serves as a testament for its self-acquisition oflexical semantics.

Furthermore, SenseBERTBASE surpasses prior systems in the Word inContext (WiC) task (Pilehvar and Camacho-Collados, 2019) from theSuperGLUE benchmark (Wang et al., 2019), which directly depends onword-supersense awareness.

In some examples, a single SenseBERTLARGE model has achieved state ofthe art performance on WiC with a score of 72.14, improving the score ofcertain prior systems by 2.5 points. For example, certain BERT modelstrained with current word-level self-supervision, burdened with theimplicit task of disambiguating word meanings, often fails to grasplexical semantics, exhibiting high supersense misclassification rates.The weakly-supervised word-sense signal used in the presently disclosedmodels. for example, may allow SenseBERT to significantly bridge thisgap.

Moreover, SenseBERT may exhibit an improvement in lexical semanticsability (reflected by the Word in Context task score) even when comparedto models with WordNet infused linguistic knowledge.

Further details regarding a method for integrating wordsense-information within SenseBERT's pre-training is described. Theinput to BERT is a sequence of words {x^(j)∈{0,1}D_(W)}_(j=1) ^(N) where15% of the words are replaced by a [MASK] token. Here N is the inputsentence length, D_(W) is the word vocabulary size and x^((th)) is aI-hot vector corresponding to the j^(th) input word. For every maskedword, the output of the pretraining task is a word-score vectory^(words)∈

^(D) ^(W) containing the per-word score. BERT's architecture can bedecomposed to (1) an internal Transformer encoder architecture wrappedby (2) an external mapping to the word vocabulary space denoted by W.

The Transformer encoder operates over a sequence of word embeddingsv_(input) ^((j))∈

^(d), where d is the Transformer encoder's hidden dimension. These arepassed through multiple attention-based Transformer layers, producing anew sequence of contextualized embeddings at each layer. The Transformerencoder output is the final sequence of contextualized word embeddingsv_(input) ^((j))∈

^(d).

The external mapping W∈

^(d×D) ^(W) is effectively a translation between the external wordvocabulary dimension and the internal Transformer dimension. Originalwords in the input sentence are translated into the Transformer block byapplying this mapping (and adding positional encoding p^((j))∈

^(d)):v _(input) ^((j)) Wx ^((j)) +p ^((j))  (1)

The word-score vector for a masked word at position j is extracted fromthe Transformer encoder output by applying the transpose:y^(words)=W^(T)v_(input) ^((j)). The use of the same matrix W as themapping in and out of the transformer encoder space may be referred toas weight tying.

Given a masked word in position j, BERT's original masked-wordprediction pre-training task is to have the softmax of the word-scorevector y^(words)=W^(t)v_(input) ^((j)) get as close as possible to a1-hot vector corresponding to the masked word. This may be done byminimizing the cross-entropy loss between the softmax of the word-scorevector and a 1-hot vector corresponding to the masked word:

_(M)=−log p(w|context),  (2)

where w is the masked word, the context is composed of the rest of theinput sequence, and the probability is computed by:

$\begin{matrix}{{{p\left( {w❘{context}} \right)} = \frac{\exp\left( y_{w}^{words} \right)}{2\; a}},} & (3)\end{matrix}$

where y_(w) ^(words) denotes the w^(th) entry of the word-score vector.

Jointly with the above procedure for training the word-level languagemodel of SenseBERT, the model may be trained to predict the supersenseof every masked word, thereby training a semantic-level language model.This may be done by adding a parallel external mapping to the wordssupersenses space, denoted S∈

^(d×D) ^(S) , where D_(S)=45 is the size of supersenses vocabulary.Ideally, the objective is to have the softmax of the sense-score vectory^(senses)∈

^(D) ^(S) :=S^(T)v_(output) ^((j)) get as close as possible to a 1-hotvector corresponding to the word's supersense in the given context.

For each word w in our vocabulary, the WordNet word-sense inventory maybe employed for constructing A(w), the set of its “allowed” supersenses.Specifically, we apply a WordNet Lemmatizer on w, extract the differentsynsets that are mapped to the lemmatized word in WordNet, and defineA(w) as the union of supersenses coupled to each of these synsets. Asexceptions, we set A(w)=Ø for the following: (i) short words (up to 3characters), because they are often treated as abbreviations, (ii) stopwords, as WordNet does not contain their main synset (e.g. ‘he’ iseither the element helium or the hebrew language according to WordNet),and (iii) tokens that represent part-of-word.

Given the above construction, a combination of two loss terms may beemployed for the supersense-level language model. The followingallowed-senses term may maximize the probability that the predictedsense is in the set of allowed supersenses of the masked word w:

$\begin{matrix}{{\mathcal{L}_{SLM}^{allowed} = {{{- \log}\;{p\left( {s \in {A\left( {w❘{context}} \right)}} \right)}} = {{- \log}\;{\sum\limits_{s \in {A{(w)}}}{p\left( {s❘{context}} \right)}}}}},} & (4)\end{matrix}$

where the probability for a supersense s is given by

$\begin{matrix}{{p\left( {s❘{context}} \right)} = {\frac{\exp\left( y_{s}^{senses} \right)}{\sum_{s^{\prime}}{\exp\left( y_{s^{\prime}}^{senses} \right)}}.}} & (5)\end{matrix}$

The soft-labeling scheme given above, which treats all the allowedsupersenses of the masked word equally, may introduce noise to thesupersense labels. We expect that encountering many contexts in asufficiently large corpus may reinforce the correct labels whereas thesignal of incorrect labels may diminish. To illustrate this, considerthe following examples for the food context:

-   -   1. “This bass is delicious”    -   (supersenses: noun.food, noun.artifact, etc.)    -   2. “This chocolate is delicious”    -   (supersenses. noun.food, noun.attribute, etc.)    -   3. “This pickle is delicious”    -   (supersenses: noun.food, noun.state, etc.)

Masking the marked word in each of the examples results in threeidentical input sequences, each with a different sets of labels. Theground truth label, noun.food, appears in all cases, so that itsprobability in contexts indicating food is increased whereas the signalssupporting other labels cancel out.

While

_(SLM) ^(allowed) pushes the network in the right direction, minimizingthis loss could result in the network becoming overconfident inpredicting a strict subset of the allowed senses for a given word, i.e.,a collapse of the prediction distribution. This is especially acute inthe early stages of the training procedure, when the network couldconverge to the noisy signal of the soft-labeling scheme.

To mitigate this issue, the following regularization term may be addedto the loss. which may encourage a uniform prediction distribution overthe allowed supersenses:

$\begin{matrix}{{\mathcal{L}_{SLM}^{reg} = {- {\sum\limits_{s \in {A{(w)}}}{\frac{1}{{A(w)}}\log\;{p\left( {s❘{context}} \right)}}}}},} & (6)\end{matrix}$

i.e., a cross-entropy loss with a uniform distribution over the allowedsupersenses.

Overall, jointly with the regular word level language model trained withthe loss in eq. 2, the semantic level language model may be trained witha combined loss of the form:

_(SLM)=

_(SLM) ^(allowed)+

_(SLM) ^(reg).  (7)

Though in principle two different matrices could have been used forconverting in and out of the Tranformer encoder, the BERT architectureemploys the same mapping W, This approach, referred to as weight tying,has been shown to yield theoretical and practical benefits. Intuitively,constructing the Transformer encoder's input embeddings from the samemapping with which the scores are computed improves their quality as itmakes the input more sensitive to the training signal.

Following this approach, and inserting our newly proposed semantic-levellanguage model matrix S in the input in addition to W, as shown in FIG.10B (contrast with FIG. 10A), such that the input vector to theTransformer encoder (eq. 1) is modified to obey:v _(input) ^((j))=(W+SM)x ^((j)) +p ^((j)),  (8)

where p^((j)) are the regular positional embeddings as used in BERT, andM∈

^(D) ^(S) ^(×D) ^(W) is a static 0/1 matrix converting between words andtheir allowed WordNet supersenses A(w).

The above strategy for constructing v_(input) ^((j)) may allow for thesemantic level vectors in S to come into play and shape the inputembeddings even for words which are rarely observed in the trainingcorpus. For such a word, the corresponding row in W is potentially lessinformative, because due to the low word frequency the model did nothave sufficient chance to adequately learn it. However, since the modellearns a representation of its supersense, the corresponding row in S isinformative of the semantic category of the word. Therefore, the inputembedding in eq. 8 can potentially help the model to elicit meaningfulinformation even when the masked word is rare, allowing for betterexploitation of the training corpus.

At the pre-processing stage, when an out-of vocabulary (OOV) word isencountered in the corpus, it may be divided into several in-vocabularysubword tokens. For the self-supervised word prediction task (eq. 2),masked sub-word tokens may be straightforwardly predicted. In contrast,word-sense supervision may be meaningful only at the word level. Wecompare two alternatives for dealing with tokenized OOV words for thesupersense prediction task (eq. 7).

In the first alternative, called 60K vocabulary, we augment BERT'soriginal 30K-token vocabulary (which roughly contained the most frequentwords) with an additional 30K new words, chosen according to theirfrequency in Wikipedia. This vocabulary increase may allow us to seemore of the corpus as whole words for which supersense prediction is ameaningful operation. Additionally, in accordance with the discussionabove, our sense-aware input embedding mechanism can help the modelextract more information from lower frequency words. For the cases wherea sub-word token is chosen for masking, we may only propagate theregular word level loss and may not train the supersense predictiontask.

The above addition to the vocabulary may result in an increase ofapproximately 23M parameters over the 110M parameters of BERTBASE and anincrease of approximately 30M parameters over the 340M parameters ofBERT_(LARGE) (due to different embedding dimensions d=768 and d=1024,respectively).

It is worth noting that similar vocabulary sizes in leading models havenot resulted in increased sense awareness. As a second alternative,referred to as average embedding, we may employ BERT's regular 30K-tokenvocabulary and employ a whole-word-masking strategy. Accordingly, all ofthe tokens of a tokenized OOV word may be masked together. In this case,the supersense prediction task may be trained to predict theWordNetsupersenses of this word from the average of the output embeddings atthe location of the masked sub-words tokens.

Words that have a single supersense may serve as good anchors forobtaining an unambiguous semantic signal. These words teach the model toaccurately map contexts to supersenses, such that it is then able tomake correct context-based predictions even when a masked word hasseveral supersenses. We therefore favor such words in the maskingstrategy. choosing. for example, 50% of the single-supersensed words ineach input sequence to be masked. We may stop if 40% of the overall 15%masking budget is filled with single-supersensed words (which rarelyhappens), and in any case the choice of the remaining words to completethis budget may be randomized. As in the original BERT. I out of 10words chosen for masking may be shown to the model as themselves ratherthan being replaced with [MASK].

A SenseBERT pretrained as described above may have an immediatenon-trivial bi-product. The pre-trained mapping to the supersensesspace, denoted S, may act as an additional head predicting a word'ssupersense given context, as shown in FIG. 10 .

A semantic-level language model may be attained that predicts themissing word's meaning jointly with the standard word-form levellanguage model. The resultant mapping is shown in FIGS. 11A-B, whichillustrate a UMAP dimensionality reduction of the rows of S. whichcorresponds to the different supersenses. A clustering according to thesupersense part of speech is apparent in FIG. 11A. Finer-grainedsemantic clusters may further be identified, as shown for example inFIG. 11B.

SenseBERT's semantic language model may allow prediction of adistribution over supersenses rather than over words in a maskedposition. FIGS. 12A-B show the supersense probabilities assigned bySenseBERT in several contexts, demonstrating the model's ability toassign semantically meaningful categories to the masked position.

Finally, we demonstrate that SenseBERT enjoys an ability to view rawtext at a lexical semantic level. FIG. 12B shows example sentences andtheir supersense predictions by the pretrained model. Where a vanillaBERT would see only the words of the sentence “Dan cooked a bass on thegrill”. SenseBERT would also have access to the supersense abstraction:“[Person][created][food] on the [artifact]”. This sense-levelperspective can help the model extract more knowledge from everytraining example, and to generalize semantically similar notions whichdo not share the same phrasing.

The disclosed models and writing assistant have been shown to offersignificant performance improvements over existing systems (e.g., basedon various standardized benchmark tests). Such performance increases maybe achieved, for example, by the introduction of lexical semanticinformation into a neural language model's pre-training objective. Thismay result in a boosted word-level semantic awareness of the resultantmodel, referred to herein as SenseBERT, which considerably outperforms avanilla BERT on a SemEval based Supersense Disambiguation task and hasachieved state of the art results on the Word in Context task. Notably,this improvement was obtained without human annotation, but rather byharnessing an external linguistic knowledge source. This work indicatesthat semantic signals extending beyond the lexical level can besimilarly introduced at the pre-training stage, allowing the network toelicit further insight without human supervision.

The systems and methods described above are presented in no particularorder and can performed in any order and combination. For example,various embodiments of the writing assistant may include a combinationof all of the features and functionality described above, or in somecases, the writing assistant may offer any subset of described featuresand/or functionality.

The above-described systems and method can be executed by computerprogram instructions that may also be stored in a computer readablemedium that can direct a computer, other programmable data processingapparatus, or other devices to function in a particular manner, suchthat the instructions stored in the computer readable medium produceinstructions which when implemented cause the writing assistant toperform the above-described methods.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the above-described methods.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from the invention described in this specification.The descriptions in this specification are for purposes of illustrationonly and are not to be construed in a limiting sense. The scope of thepresent invention is limited only by the language of the followingclaims.

What is claimed is:
 1. A non-transitory computer readable mediumincluding instructions that when executed by one or more processingdevices cause the one or more processing devices to perform a methodincluding: initiating a writing assistant application in response toinput received from a user, the writing assistant application beingassociated with at least one graphical user interface element shown on adisplay; automatically analyzing, using one or more trained modelsproviding a natural language generation function, at least one textelement and identifying one or more contextual elements associated withthe at least one text element; automatically constructing, using the oneor more trained models, at least one complete sentence textual outputoption that differs from the at least one text element in at least onerespect, expresses a meaning associated with the at least one textelement, and agrees with at least one of the one or more contextualelements identified relative to the at least one text element; andcausing the at least one complete sentence textual output option to beshown to the user via the display.
 2. The mcthod non-transitory computerreadable medium of claim 1, wherein the at least one text element isincluded as part of an electronic document.
 3. The non-transitorycomputer readable medium of claim 2, wherein the electronic documentincludes an email document.
 4. The mcthod non-transitory computerreadable medium of claim 1, wherein the at least one complete sentencetextual output option includes two or more complete sentence textualoutput options selectable by the user for insertion into an electronicdocument.
 5. The non-transitory computer readable medium of claim 1,wherein the at least one complete sentence textual output option isconstructed for insertion into an electronic document as a possiblereplacement for the at least one text element.
 6. The non-transitorycomputer readable medium of claim 1, wherein the method furtherincludes: receiving user input, wherein the user input includes at leastone word; retrieving information from an external source, based on atleast one attribute associated with the user input; automaticallyconstructing at least one textual output option that conveys theretrieved information and expresses a meaning associated with the userinput; and causing the at least one textual output option to be shown tothe user via the display.
 7. The non-transitory computer readable mediumof claim 1, wherein the method further includes: receiving user input,wherein the user input includes a collection of two or more words thatconvey at least one idea and one or more facts; retrieving informationfrom an external source, based on the one or more facts included in thecollection of two or more words; automatically constructing at least onecomplete sentence option that expresses the at least one idea andconveys the one or more facts, wherein the at least one completesentence is also automatically constructed to be consistent with theinformation retrieved from the external source; and causing the at leastone complete sentence option to be shown to the user via the display. 8.The non-transitory computer readable medium of claim 1, wherein themethod further includes: in response to the initiation of the writingassistant application, causing a writing assistant workspace to be shownon the display; receiving primary user input, wherein receipt of theprimary user input is facilitated by the writing assistant workspace;based on at least one attribute of the primary user input, causing aprimary structured input template to be shown on the display; receivingsecondary user input via the primary structured input template, whereinthe secondary user input conveys information with respect to at leastone predetermined subject associated with the primary structured inputtemplate; automatically constructing at least one complete sentenceoption that references the predetermined subject and includes theinformation conveyed by the secondary user input; and causing the atleast one complete sentence option to be shown to the user via thewriting assistant workspace on the display.
 9. The non-transitorycomputer readable medium of claim 1, wherein the method furtherincludes: in response to the initiation of the writing assistantapplication, causing a writing assistant workspace to be shown on thedisplay; receiving primary user input, facilitated by the writingassistant workspace, wherein the primary user input includes acollection of two or more words that convey at least one idea;determining, based on analysis of the primary user input, at least oneinformation item not conveyed by the primary user input; prompting theuser, via the writing assistant workspace, to enter a secondary userinput associated with the at least one information item not conveyed bythe primary user input; receiving the secondary user input via astructured input template, wherein the secondary user input includes oneor more informational details associated with the at least oneinformation item not conveyed by the primary user input; automaticallyconstructing at least one complete sentence option that expresses the atleast one idea and conveys the one or more informational detailsincluded with the secondary user input; and causing the at least onecomplete sentence option to be shown to the user via the writingassistant workspace on the display.
 10. The non-transitory computerreadable medium of claim 1, wherein the at least one complete sentencetextual output option includes two or more complete sentence textualoutput options, and wherein the method further includes: causing the twoor more complete sentence textual tcxt output options to be shown on thedisplay; receiving from the user an indication of a selection of one ofthe two or more tcxt complete sentence textual output options;generating one or more refined text output options based on the selectedone of the two or more complete sentence textual output options; andcausing the one or more refined text output options to be shown on thedisplay.
 11. The non-transitory computer readable medium of claim 1,wherein the method further includes: receiving from the user anindication of a text insertion location in an electronic document;generating the at least one complete sentence textual output option forinsertion at the text insertion location in the electronic document,wherein the at least one complete sentence textual output option linksat least one aspect of a first text element that precedes the textinsertion location with a second text element that follows the textinsertion location; and causing the at least one complete sentencetextual output option to be shown to the user via the display.
 12. Thenon-transitory computer readable medium of claim 1, wherein the one ormore trained models are trained machine learning models.
 13. Thenon-transitory computer readable medium of claim 1, wherein the writingassistant application is incorporated into word processing software, anemail editor, or presentation software.
 14. The non-transitory computerreadable medium of claim 1, wherein the at least one complete sentencetextual output option is shown on the display in association with acontrol element configured to cause the at least one complete sentencetextual output option to be inserted into an electronic document as areplacement for the at least one text element.
 15. The non-transitorycomputer readable medium of claim 1, wherein the method further includescausing at least one control element to be shown on the display, whereinthe at least one control element is configured to control a formalitylevel of the at least one complete sentence textual output option. 16.The non-transitory computer readable medium of claim 1, wherein themethod further includes causing at least one control element to be shownon the display, wherein the at least one control element is configuredto control a length of the at least one complete sentence textual outputoption.
 17. The non-transitory computer readable medium of claim 1,wherein the at least one graphical user interface element includes atleast one of a window, a field, a virtual button, an icon, or a menuitem.
 18. The non-transitory computer readable medium of claim 1,wherein the at least one text element includes a group of wordscomprising a partial sentence or a sentence fragment.